Atomic EEG

Towards Global-scale Species Distribution Modelling

2025-04-11T18:19:28Z

Abstract: Estimating the geographical range of a species from sparse observations is a challenging and important geospatial prediction problem. Given a set of locations where a species has been observed, the goal is to build a model to predict whether the species is present or absent at any location. This problem has a long history in ecology, but traditional methods struggle to take advantage of emerging large-scale crowdsourced datasets which can include tens of millions of observations of hundreds of thousands of species in addition to the availability of multi-modal data sources such as paired images and natural language descriptions. In this talk, I will present recent work from my group where we have developed deep learning-based solutions for estimating species' ranges from sparse...

Dell R640 installation

2025-04-10T01:00:00Z

Today we have racked the five 14th generation Dell R640 servers and a Dell N4032 switch. When inspecting the rack rails, I noticed that some of the left-hand rails had an extra tab on them while the others did not. For the first server, I used a rail with a tab only to discover that the tab prohibited the server from being pushed in all the way. The tabs were easily removed but the server needed to be removed from the rack first. First server installed The last server on the rails Front view Rear view Cabling Yellow CAT5 for iDRAC ports Red CAT6 for 10GBase-T The initial iDRAC configuration was carried out using a crash cart. The servers are called: myrina thalestris lampedo otrera antiope We had some difficulty with the 40G uplink from the switch and we could only get the link to come up...

Weeknotes 2025-W15

2025-04-09T13:53:56Z

I have a lot to say this week, so strap in. Forester 5.0 development: canonical URLs, atom feeds, and LSP Work on Forester 5.0 proceeds apace, generously supported by ARIA who have engaged Kento Okura and myself on a consulting basis to support their internal use of Forester. My recent goals have been to bring Forester closer in line with the architecture of the World Wide Web; to that end, I have made two big improvements. First cut at canonical URLs Trees are addressed by “canonical URLs” that are meant to be the place where they will ultimately be published. See my blog post on the design for more details. Canonical URLs are of the form https://www.my-host.net/tree-name/ ; the handling is a little fragile right now and you can expect bugs (but please write to me about them). First cut...

Coping and Capping

2025-04-08T01:00:00Z

Well that was fun! Quite high up there in the set of things that I never even considered I might do would’ve been awarding degrees. But by dint of being President and thus standing in for Simon , on Saturday I did exactly that. The University held a Congregation for those being admitted to the Cambridge MA (“Magistri in Artibus” – Master of Arts). Degrees are conferred by the Chancellor, Vice-Chancellor or nominated deputy. Apparently, typically, for this one and the main undergraduate congregation in July, that nominated deputy Vice-Chancellor is usually the Head of House for the College concerned. In this case, as President at Christ’s is effectively deputy Master (~ Head of House), it me. So instead of the usual Batman-style black gown, I got to wear the rather natty Superman-style...

Weeknotes: 7th April 2025

2025-04-07T10:07:12Z

Last Week Talking to new team members We welcomed Mark , David , and Jon to the group recently, and I finally found time to sit down and have some great discussions with them all on what we've been working on this last year, and how we might all collaborate on things around that. In what was an otherwise frustrating week on the technical work front, it was great to have some inspiring discussions on what we might work on over the coming year. Life A wise robot once said: Life! Don't talk to me about life. And I've held a certain amount of sympathy for that view of late, it has to be said, as I've been fighting with the LIFE pipeline as I try to run some new high-resolution maps for parts of the globe. This week has been plagued by what I felt was reasonable decision the...

Box API with OCaml and Claude

2025-04-07T01:00:00Z

Over the weekend, I decided to extend my Box tool to incorporate file upload. There is a straightforward POST API for this with a curl one-liner given in the Box documentation . Easy. The documentation for Cohttp-eio.Client only gives the function signature for post , but it looked pretty similar to get , which I had already been working with. The README for Cohttp gave me pause when I read this comment about multipart forms. Multipart form data is not supported out of the box but is provided by external libraries Of the three options given, the second option looked abandoned, while the third said it didn’t support streaming, so I went with the first one dionsaure/multipart_form . The landing page included an example encoder. A couple of external functions are mentioned, and I found example code...

2025-04-06 Weekly Notes

2025-04-06T01:00:00Z

As mentioned in my previous post, I have been working on the Gini Coefficient. However, when trying to run the code at a different geographical scale, it crashed, so I decided to reformat the code into a module following software engineering practices, which has been my sole focus the past week. I realised I had written the same function in multiple scripts, making the code redundant. This resulted in the reduction of the number of files and depedencies now are more clear. I switched the python project from an entirely geopandas-based workflow to a an Apache Sedona-based one, which gives me more flexibility and efficiency. I added a couple of new features that enable the location of trees around multiple buffer for each building in England and I also calculated the euclidean distance...

Semi distributed filesystems with ZFS and Sanoid

2025-04-05T01:00:00Z

Over in my EEG group, we have a lot of primary and secondary datasets lying around: 100s of terabytes of satellite imagery , biodiversity data , academic literature , and the intermediate computations that go along with them. Our trusty central shared storage server running TrueNAS stores data in ZFS and serves it over NFSv4 to a bunch of hosts. This is rapidly becoming a bottleneck as our group and datasets grow, and Mark Elvers has been steadily adding lots more raw capacity . The question now is how to configure this raw SSD capacity into a more nimble storage setup. If anyone's seen any systems similar to the one sketched out below, I'd love to hear from you. Why get rid of NFS? The first design constraint is to get rid of centralised network storage. This is both slow when compared to a modern...

Rewarding influence

2025-04-05T01:00:00Z

EuroSys 2025 was co-located with ASPLOS 2025 this year. Other commitments meant I (again) couldn’t stay for the whole conference, attending primarily because two students had papers in the TDIS workshop . But happily I was able to stay for the first day of the conference – “happily” not only because it gave me a chance to catch up with some old friends I hadn’t seen in a decade or more, but also because the Mirage unikernels paper which appeared at ASPLOS 2013 won one of two ASPLOS 2025 Influential Papers awards :) This is obviously very flattering – typically doing research is necessarily its own reward because the work can seem fruitless much of the time. Even when a paper gets written and submitted it will most likely be rejected – I think EuroSys this year reported something like a 12% acceptance rate, so rejection is a priori the most likely outcome. Finally, if...

opam repo ci job timeouts

2025-04-04T01:00:00Z

It’s Tuesday morning, and virtually all opam repo ci jobs are failing with timeouts. This comes at a critical time as these are the first jobs following the update of ocurrent/ocaml-version noted on 24th March. The opam repo ci tests all PRs on opam-repository . The pipeline downloads Docker images, which contain the root filesystem for various Linux distributions, architectures, and OCaml versions, which are used as the base environment to run the tests. These base images are created by the base image builder . PR#317 update these base images in three ways: Images for OCaml < 4.08 were removed. The opam-repository-archive overlay was removed as this contained the < 4.08 opam packages. The ocaml-patches-overlay overlay was removed as this was only needed to build OCaml < 4.08 on GCC 14. Given these...

2nd Programming for the Planet workshop CFP out

2025-04-03T01:00:00Z

Dominic Orchard and I had a blast running the first PROPL workshop a couple of years ago, with a full room and engaged audience in POPL in London. Last year, our sister conference LOCO took over, and it's our turn again this year! PROPL will return for a second outing in October, co-located with ICFP /SPLASH in Singapore in October. Read the call for papers here (deadline 3rd July 2025). Dominic prepping for the first PROPL in the rather delightful venue We'd love to get wider participation in computer science interacting with matters of climate and biodiversity: There are simultaneous interlinked crises across the planet due to human actions: climate change, biodiversity loss, and desertification. Addressing these challenges requires, amongst other things, a global understanding...

Everything is a file; except when it’s not

2025-04-03T01:00:00Z

Some titles make more sense than others. One of my oldest contributions to OCaml was a complete overhaul of Unix.stat et al in ocaml/ocaml#462 which formed part of OCaml 4.03. As part of the work on msvs-detect in late 2015, I’d ended up with a Windows 7 VM which had every single version of Visual Studio back to Visual Studio 6.0 . Visual Studio (and Visual C++ before that) has always included the source code for the C Runtime Library (CRT), and as a side-effect of having all these installed Visual Studios, I was able to construct a Git repository showing the evolution of the CRT code over each release (sadly, the licence doesn’t allow this to be pushed publicly). This was particularly useful for studying how the behaviour of the stat implementation had changed over time, particularly with reference to...

More Kingston Drives

2025-04-03T01:00:00Z

We have received the second batch of 40 x 7.68TB Kingston SSD drives, bringing the total to 50 drives. We now have 5 fully populated Dell PowerEdge R640 with a total raw capacity of 384TB.

Ubuntu with ZFS root

2025-04-02T01:00:00Z

The installation of Ubuntu on ZFS contains about 50 steps of detailed configuration. I have 10 servers to install, so I would like to script this process as much as possible. To test my script, I have created a new VM on VMware ESXi with 10 x 16GB disks, 16GB RAM, 4 vCPU. In the advanced options, I have set the boot to EFI and set disk.EnableUUID = "TRUE" in the .vmx file. Doing this ensures that /dev/disk aliases are created in the guest. Boot Ubuntu 24.04 from the Live CD and install SSH. sudo -i apt update apt install openssh-server -y Use wget to download https://github.com/mtelvers.keys into ~/.ssh/authorized_keys . wget https://github.com/mtelvers.keys -O ~/.ssh/authorized_keys In your Ansible hosts file, add your new machine and its IP address your.fqdn ansible_host= Run the...

Updating Docker and Go

2025-04-01T01:00:00Z

For some time, we have had issues on Ubuntu Noble when extracting tar files within Docker containers. See ocaml/infrastructure#121 . This is only an issue on exotic architectures like RISCV and PPC64LE. # docker run --rm -it ubuntu:noble root@cf3491db4abd:/# cd root@cf3491db4abd:~# mkdir foo root@cf3491db4abd:~# tar -cf bar.tar foo root@cf3491db4abd:~# rmdir foo root@cf3491db4abd:~# tar -xf bar.tar tar : foo: Cannot change mode to rwxr-xr-x: Operation not permitted tar : Exiting with failure status due to previous errors The combination of Docker version and libseccomp2 version prevents the container from running the fchmodat2 system call. There is a bug report on Ubuntu’s bug tracker for the issue. I have been working around this by building Docker from scratch. apt install golang git clone...

Weeknotes: 31th March 2025

2025-03-31T10:07:12Z

Last Week I was still coming off the back of being unwell the previous week, so not the most productive week STAR I got the full STAR calculation running, which means I now have two full biodiversity impact pipelines running built on the same AoH code underpinnings. I checked in with Chess, and it looks like I'm mostly there, but I just want to check some of the details, as it looks like I'm losing some costal pixels compared to her results, which makes me suspect my clipping for the marine layer might be out. LIFE I started to process a bunch of high-resolution data I've been given for brazil to make an application specific version of the LIFE metric . My intent had been to try and code something to make it more efficient to work with a mix of high and low resolution maps, rather than doing a...

Shelter, Hazel and More!

2025-03-31T01:00:00Z

Last week I focused on Shelter -- our idea that shells should have the same ability as reproducible build tools like Nix or Docker. To this end I now have a fairly fleshed out prototype. Shelter Prototype Shelter is a spin-off from the work Michael and I started with Shark . It takes the same ideas but applies them directly to a shell-like interface. We're still in the middle of working all of this, but you can read more about it at Shelter . Forester Hacking As you can probably tell, my website is still using Forester . I rebased my Markdown branch to include the new Atom syndication feature. Alongside that I added support for arbitrary HTML injection into Forester via codeblocks in Markdown. This was actually very straightforward thanks to Markup and being able to...

Installation order for opam packages

2025-03-31T01:00:00Z

Previously, I discussed the installation order for a simple directed acyclic graph without any cycles. However, opam packages include post dependencies. Rather than package A depending upon B where B would be installed first, post dependencies require X to be installed after Y. The post dependencies only occur in a small number of core OCaml packages. They are quite often empty and exist to direct the solver. Up until now, I had been using a base layer with an opam switch containing the base compiler and, therefore, did not need to deal with any post dependencies. Here is the graph of 0install with post dependencies coloured in red. Removing the post dependencies gives an unsatisfying graph with orphaned dependencies. 0install without post . Note base-nnp.base and base-effects.base . However, this...

2025-03-30 Weekly Notes

2025-03-30T00:00:00Z

The past week or so I’ve been working on cleaning the code in my 3-30-300 analysis repo. Most of the code was running fine but many functions were repeated in the modules, and it wasn’t really structured as a package. This was partially motivated by a workshop about good coding practices by the photonics group in the department of electrical engineering. Having a clean repo makes all the difference when running your experiments. In my meeting with Anil we discussed on the best ways to measure the Gini coefficient for green space accessibility and availability and how they relate to equity. Although, widely used in econometrics, it presents some problems when it comes to scale. For this reason, I am calculating the 3-30-300 metrics and the remote sensing-derived metrics at the...

Not one but two accepted papers and a Ph.D.

2025-03-30T00:00:00Z

As I find myself once more on a train to parts unknown (to me at least), a brief update :) The parts unknown in question is Rotterdam, NL (so really quite well-known to quite a lot of people, just not me) for EURO/SYS 2025 (being held jointly with ASPLOS 2025 , although I can’t stay for the whole thing unfortunately) and specifically the 3rd International Workshop of Testing Distributed Internet of Things Systems (TDIS) . Why? Happily the programme committee decided to accept two papers from my (ex-)students – which is nice :) The two in question are Reckon-ing Kubernetes at the Edge using Emulated Clusters with Alessandro Sassi (University of Cambridge / Politecnico di Milano) and Christopher Jensen (University of Cambridge / Microsoft Research). This describes Alessandro’s M.Sc. research project undertaken as a visitor with my group. He built on Chris’ earlier work on Reckon, an emulator setup for examining consensus system behaviour . Alessandor extended this to...

Box Diff Tool

2025-03-30T00:00:00Z

Box has an unlimited storage model but has an upload limit of 1TB per month. I have been uploading various data silos but would now like to verify that the data is all present. Box has an extensive API , but I only need the list items in folder call. The list-items call assumes that you have a folder ID which you would like to query. The root of the tree is always ID 0. To check for the presence of file foo in a folder tree a/b/c/foo , we need to call the API with folder ID 0. This returns a list of entries in that folder. e.g. { "entries" : [ { "id" : "12345" , "type" : "folder" , "name" : "a" } ] } The API must now be called again with the new ID number to get the contents of folder a . This is repeated until we finally have the entries for folder c which would contain the file itself. I...

Blockchain-Based Carbon Trading for Stakeholders in Brazil

2025-03-28T17:09:11Z

Full Title: Democratizing Carbon Markets: A Blockchain-Based Emission Trading System for Small and Large-Scale Stakeholders in Brazil Abstract: The integration of blockchain technology into carbon markets offers a unique opportunity to create more transparent, inclusive, and efficient trading mechanisms. This presentation introduces a novel Blockchain Emission Trading System (BETS) model designed to align with Brazil’s new carbon market legislation (Law 15042/2024), ensuring that both large landholders and small rural producers can participate fairly. Our approach leverages official land registries, such as SICAR, to create spatially and temporally verifiable carbon credits, preventing fraud and double counting while enabling greater accessibility for smaller stakeholders who often...

New preprint on benchmarking ultra-low power neural accelerators

2025-03-28T00:00:00Z

Josh Millar just released our latest preprint on how to make sense of the growing number of dedicated, ultra-low-power 'neural network accelerators' that are found in many modern embedded chipsets. My interest in this derives from wanting to decouple from the cloud when it comes to low-latency local environments , and this needs fast tensor operations in hardware. Josh found a huge number of interesting NPUs in modern low-cost chips, ranging from ESP32 -based boards over to ARM ones. All of these have quite a variety of tradeoffs, from the operations supported (which affects which models can be run on them) to the amount of memory and CPU power. Josh Millar , Yushan Huang , Sarab Sethi , Hamed Haddadi and Anil Madhavapeddy . Working paper at arXiv . URL (arxiv.org) DOI BIB...

OCaml Core Dev Meeting

2025-03-28T00:00:00Z

OCaml Core Dev meeting at Inria yesterday. These are roughly biannual synchronous catchups which provide a chance to find out what others are up to, get feedback on any major ongoing work, and attempt to unblock some stalled PRs. Sometimes tempers get frayed, but not this time round… Quick round-table: ongoing work to get modular explicits/implicits into OCaml; a lot of work on GC pacing to get the 5.x GC to follow its space overhead setting properly; lot of work on flambda 2; OCaml Language committee has been launched and has two proposals under consideration; quite a lot of reviewing of new features from Jane Street in flight; work on pluggable GC to be able to try out alternate GCs in the main runtime. Two presentations on the table. The OCaml 5.x GC isn’t working hard enough to...

Dell PowerEdge R640 Storage Server

2025-03-27T00:00:00Z

We have received our first batch of 7.68TB Kingston SSD drives for deployment in some Dell PowerEdge R640 servers, which will be used to create a large storage pool. The first job was to mount each of the drives in a caddy. And then install them in the server. These R640 servers are equipped with the Dell PERC H740P RAID controller. They support either hardware RAID 0,1,5,10,50 etc or Enhanced HBA mode. In eHBA mode, the disks operate in a passthrough mode, presenting the raw disks to the OS, however each disk needs to be specifically selected in an additional step after enabling eHBA mode. In RAID mode, one or more virtual disks need to be created to present the disks to the OS. Preconfigured profiles are available to complete this step easily. We will run these with a ZFS file system,...

How This Site Is Build

2025-03-26T00:00:00Z

Published Wed 26 Mar 2025. This site has continuously evolved since I made the first commit while procrastinating my undergrad dissertation, __LINK_0__ commit 632cb1f0c97c07fb99b48192444397e56ea5310f __LINK_1__ Author: Ryan Gibb __LINK_2__ Date: Fri Jan 22 11:27:55 2021 +0000 __LINK_3__ __LINK_4__ Initial commit __LINK_5__ __LINK_6__ diff --git a/index.html b/index.html __LINK_7__ new file mode 100644 __LINK_8__ index 0000000..557db03 __LINK_9__ --- /dev/null __LINK_10__ +++ b/index.html __LINK_11__ @@ -0,0 +1 @@ __LINK_12__ +Hello World I started off writing plain HTML, then switching to writing in markdown and using pandoc to convert to HTML, and gradually accumulated bash scripts and makefiles to add more functionality, such as generating an Atom feed . This became...

FreeBSD 14.2 Upgrade

2025-03-26T00:00:00Z

CI workers spring and summer run FreeBSD and need to be updated. Check the current version of FreeBSD which we have with uname -r . FreeBSD summer 14.1-RELEASE-p5 FreeBSD 14.1-RELEASE-p5 GENERIC amd64 Run freebsd-update fetch to download the latest versions of the system components, particularly the freebsd-update utility. It even reported that it really is time to upgrade! # freebsd-update fetch ... WARNING: FreeBSD 14.1-RELEASE-p5 is approaching its End-of-Life date. It is strongly recommended that you upgrade to a newer release within the next 5 days. Install these updates. freebsd-update install Now use freebsd-update to fetch the 14.2-RELEASE and install it. # freebsd-update upgrade -r 14.2-RELEASE ... # freebsd-update install src component not installed, skipped Installing...

Towards Forester 5.0 II: a design for canonical URLs

2025-03-25T13:44:56Z

One of the goals of Forester 5.0 is lightweight federation —the ability to have two forests participate in the same graph and therefore provide backlinks, etc. In a previous post ( Towards Forester 5.0: a design for global identity ), I talked about some of the difficulties that arise when dealing with identities of people and references that have global scope but could nonetheless be described by trees in many forests. I proposed that such things should be addressed by canonical URIs (e.g. DIDs, DOIs, etc.) and that Forester should grow the ability to bind a canonical URI to multiple trees, which are then gathered into a disambiguation page. Today I want to broaden the discussion to cover the difficulties of addressing trees themselves (as opposed to the global entities they may describe). This is a proposal and I welcome...

Weeknotes: 24th March 2025

2025-03-24T09:07:12Z

Last week I spent most of the week with a head-cold or such (I'm not the right kind of doctor to figure that out), and so productivity was a little low. STAR Two major steps forward on my implementation of the IUCN's STAR metric this week. Firstly, my extraction of species from the IUCN Red List database finally aligns with that of the official implementation in R that is built by Chess Ridley at Newcastle University. Chess has been helping me navigate the nuances of the Red List database for the last couple of months as either of us gets time to nudge things forward, and we finally got to parity this week. Secondly, I took a first pass at implementing the post-AoH stages of the STAR pipeline, (see "STAR Calculation" in the Methods section of the original paper ). This involves weighting all the...

Mon 24 Mar 2025

2025-03-24T00:00:00Z

Previous: Mon 17 Mar 2025 Babel TODO cargo dynamic loading At the moment this loads the whole index at once which takes a long time, so I need to figure out how to make pubgrub_cargo dynamically load packages. TODO add Nix solver Using the same techniques as opam-nix-repository TODO polish up babel paper with motivating examples TODO read up on answer set programming https://pubgrub-rs-guide.pages.dev/internals/intro is a good starting point And ‘Conﬂict-driven answer set solving: From theory to practice’ [1] deployment TODO Nix TODO Docker Research notes Contextualising my research activities to date. TODO Eilean TODO Babel [1] M. Gebser, B. Kaufmann, and T. Schaub, “Conflict-driven answer set solving: From theory to practice,” Artificial Intelligence , vol. 187–188, pp. 52–89, Aug. 2012, doi: 10.1016/j.artint.2012.04.001 . [Online]. Available:...

2025-03-23 Weekly Notes

2025-03-23T00:00:00Z

After the issues and errors with generating the tree data for all of England, I realised that one of the best ways to proof-check the data was to visualize it and more importantly, to let others see it so they can check things that I might be missing. Because of this, I spent a good chunck of time trying to create a web map of all the trees in London. I used CARTO to do it but it wasn’t straightforward, mostly because of the size of the data. My files were stored in parquet format which apparently CARTO nor Mapbox are capable of transforming automatically to web Mercator, so I had to change the CRS in Sedona and re-generate the parquet partitions. Unfortunately, knowing where each partition is pointed at is not very easy, so what I did was geo-reference using the UK National Grid and then...

Global Maps of Human Threats to Biodiversity and Species Extinctions

2025-03-21T16:11:39Z

Full title: Towards Global Maps of Anthropogenic Threats to Biodiversity and Their Contributions to Species Extinctions Abstract: Species extinctions are primarily driven by loss of habitat, which is relatively easy to monitor by satellite remote sensing; other anthropogenic threats to biodiversity, like hunting, are much more difficult to observe directly. My PhD project draws on local studies which capture the population effect of some anthropogenic threat, scaling these results using machine learning and remote sensing. In this talk, I will discuss my first attempt at this through quantifying species-specific responses to hunting pressure. I find that machine learning methods can offer marked improvements over (linear) statistical models, which are commonly used in ecology, but model...

LIFE becomes an Official Statistic of the UK government

2025-03-21T00:00:00Z

Our recently published LIFE biodiversity metric has just been integrated into a newly recognised Official Statistic from the UK government ! This integrates the core LIFE biodiversity metric with food provenance data to track the environmental impacts of our consumption habits. I must admit that I'd not heard of "Official Statistics" before this, so I did a bit of research. The UK Office for Statistics Regulation says that: Official statistics are statistics produced by Crown bodies and other organisations listed within an Official Statistics Order, on behalf of the UK government or devolved administrations. They provide a factual basis for assessment and decisions on economic, social and environmental issues at all levels of society. -- OSR Policies The good folk at the Joint Nature Conservation Committee are responsible for this particular statistic. The JNCC launched their...

Weeknotes: 17th March 2025

2025-03-17T09:07:12Z

Förrförra vecka Life maps As the outputs of the LIFE project we generate two maps: one showing the impact on extinction risk of converting land to farmland (aka the arable scenario), and another showing the risk of converting human changed land back to predicted its natural state (aka the restore scenario). Those maps though are data maps, containing fractional values in each pixel, so aren't something you can see, so instead when presenting the work I tend to lead with a "pop-science" version generated by QGIS when you load the maps. Here you can see the map for the arable scenario, showing that putting a farm in a tropical forest is a sub-optimal idea: And the restore scenario map here, which has less change visible, as there is less area to convert: The different colours are from...

Mon 17 Mar 2025

2025-03-17T00:00:00Z

Previous: Mon 10 Mar 2025 Next: Mon 24 Mar 2025 Babel DONE Model Context Protocol Added a basic Model Context Protocol server for querying dependencies with Babel, since LLMs seem to be particularly bad at managing dependencies. I need to flesh this out with support for metadata queries and solving for platforms. Once this works we could look at invoking ecosystem tooling to provide the dependencies, and creating a containerisation environment based on the solved platform for system dependencies. ecosystems DONE fix cross-ecosystem version ranges I had to add an abstract BabelVersionSet to interoperate pubgrub_cargp with the other dependency providers, but it doesn’t yet have the concept of empty and any ranges. Done here . Research notes Contextualising my research activities to date. DONE...

2025-03-16 Weekly Notes

2025-03-16T00:00:00Z

Most of my week was devoted to debugging my code to count the number of trees per geographical area. As mentioned in previous posts, I’ve been using Apache Sedona to process all my vector files, which has reduced the computation time significantly, particularly for spatial joins around all buildings in England. However, I realised that I hadn’t actually counted how many trees there are in total, which sounds trivial, but it turned out to be not as easy with the code that I had. So it turns out that when you are doing a Spatial Join with an Spatial RDD, you are only expected to pass two columns, the identifier and geometry for each row. This is not present in the docs, but I found it by accident while converting from Spark DataFrames to RDDs. Also, due to the number of trees (in the...

Identifying Key Countries in the Illegal Elephant Ivory Trade Networ

2025-03-14T17:34:06Z

Abstract: Illegal wildlife trade is a key driver of biodiversity loss, but targeting policy to maximise disruption to trade remains a key challenge. A network approach was applied to seizure data to prioritise national action disrupting the illegal trade of elephant ivory. By simulating the removal of countries from trade, targeting groups of countries was found to be most effective due to network redundancy. Despite temporal variability, trade was highly concentrated and cessation in less than 10 countries would have disrupted 75% of trade in 2018-2020. These findings support evidence-based legislation and efficient allocation of conservation resources for tackling illegal wildlife trade. Bio: Jakob is a PhD student in the Conservation and Development Lab (Department of Geography). His...

folding

2025-03-14T10:21:00Z

I asked my tame AI: what if the plot unfolded like a bicycle? That's a fun thought experiment! Here's how a plot might unfold like a bicycle: Two Wheels, Two Main Characters: The story revolves around two central figures, perhaps with contrasting personalities or backgrounds. They are interdependent, like the two wheels of a bicycle. One might be impulsive and adventurous, while the other is cautious and grounded. The Chain as Conflict: A central conflict or series of challenges acts as the "chain" driving the narrative forward. This could be an external threat, an internal struggle, or a shared goal. The Frame as Structure: The overall plot structure resembles the bicycle's frame – a sturdy foundation with distinct parts. There would be a clear beginning (the initial assembly), a middle...

A fully AI-generated paper just passed peer review; notes from our evidence synthesis workshop

2025-03-12T00:00:00Z

Access to reliable and timely scientific evidence is utterly vital for the practise of responsible policymaking, especially with all the turmoil in the world these days. At the same time, the evidence base on which use to make these decisions is rapidly morphing under our feet; the first entirely AI-generated paper passed peer review at an ICLR workshop today. We held a workshop on this topic of AI and evidence synthesis at Pembroke College last week, to understand both the opportunities for the use of AI here, the strengths and limitations of current tools, areas of progress and also just to chat with policymakers from DSIT and thinktanks about how to approach this rapidly moving area. (The following notes are adapted from jottings from Jessica Montgomery , Sam Reynolds , Annabelle Scott and myself. They are not at all complete, but hopefully useful!) We...

ancient history

2025-03-11T14:53:00Z

I was reminded of two things from the late 70s that a friend in cambridge did, and my cousin in london were doing at the start of their graduate research work. The first was a study of medieval court records in East Anglia, trying to figure out what the demographcs were (in terms of single.married, how many times, how many kids, ages etc) - apparently for every day people in medieval tiems there were virtually no day-to-day records execpt that most people would at some point in their lives, show up in a court...so assuming most people didn't move about much in those day,s you could get a picture by looking at statistics and comparing villages/towns etc Rhe second was a student of cave paintings from rather longer ago - the caves in France and Spain have depictions of animals from 17,000...

Weeknotes: 10th March 2025

2025-03-10T09:07:12Z

Last week iNaturalist I've continued to do a little playing around with iNaturalist data, both the range data and the occurrence data for birds. Birds, aka Aves, is a good one to pick as it's fairly well represented in both IUCN data (which I understand to have all known species of bird in it), and well covered in observations. I read up on how the "range" maps that iNaturalist have started to share , which they describe in this blog post , and link to this paper on the methods use. This describes how they use a neural network model to generate the data, and how it differs from standard modelling approaches as they only have presence data and not absence data. Anil then pointed me at a newer paper by the same group that looks specifically at trying to model species with very few occurrence points...

Mon 10 Mar 2025

2025-03-10T00:00:00Z

Previous: Mon 3 Mar 2025 Next: Mon 17 Mar 2025 Babel KILL python pubgrub Using Uv’s dependency provider . This isn’t feasible at present, as Uv implements a custom resolver. DONE cargo pubgrub Got a cargo dependency provider working modified from a repository used to benchmark pubgrub-rs on solving the cargo repository. (Root, ) -> (Cargo Bucket:serde@1.x.y, SemverPubgrub { norml: 1.0.219, pre: ∅ } ) (Cargo Bucket:serde@1.x.y, 1.0.219) -> (Cargo Bucket:serde_derive@1.x.y, SemverPubgrub { norml: >=1.0.219, <1.0.220, pre: ∅ } ), (Cargo Bucket:serde_derive@1.x.y/default=true, SemverPubgrub { norml: >=1.0.219, <1.0.220, pre: ∅ } ) (Cargo Bucket:serde_derive@1.x.y, 1.0.219) -> (Cargo Bucket:quote@1.x.y/proc-macro, SemverPubgrub { norml: >=1.0.35, <2.0.0, pre: ∅ } ), (Cargo Bucket:syn@2.x.y/parsing, SemverPubgrub { norml:...

2025-03-09 Weekly Notes

2025-03-09T00:00:00Z

Hello! This week I worked on the optimisation of counting trees within a given geometry. This is a problem I had solved back in December, although not in the optimal way. Ihad segmented almost 400M trees from the Defra dataset, but now I had to count how many are within a 100 m radius of each one of the 29M buildings in England. I had solved this by iterating through the LSOAs (statistical geography) in England, but that process took about 11 days to complete with the SQl API of Apache Sedona. When I was developing that part of my project, I had tried to solve it using Spatial RDDs, as suggested by Amelia Holcomb initially, but sadly, the documentation of some of the features in Sedona are still in developing stages so I put that away. Because I had created an improved version of the trees...

Towards Forester 5.0: a design for global identity

2025-03-08T12:17:14Z

As we move closer to Forester 5.0 , which introduces rudimentary federation capabilities, we must address new problems that did not arise in the days when no two forests interacted or linked to each other. The most immediate issue is that trees describing entities with “global” identity (including actual people as well as bibliographic references) will naturally be duplicated across many forests. For example, this happens when one person authors trees in multiple forests, and it happens even more often with bibliographic entries (both for the entries themselves and their author attributions). It is very important to handle this problem properly now in a way that (1) minimises friction and (2) enables us to quietly evolve toward more Web-centric approaches to identity as they emerge . Below, I survey some existing approaches to...

Socially self-hosting source code with Tangled on Bluesky

2025-03-08T00:00:00Z

I've been an avid user of GitHub since its launch, and it really has revolutionised how communities come together to work on open source. In recent years though, I find myself utterly overwhelmed by its notifications and want to experiment with alternative workflows . This experimentation also has a more serious undertone due to the increasing need for data sovereignty and so I'm starting to move my source code to self-hosted solutions that are less reliant on centralised services. This has also come up persistently over the years in the OCaml community, with questions over why participation in packaging requires a GitHub account ever since the early days of opam. I've never found a good answer... until now, with the launch of an exciting new service that's built over the same protocol that Bluesky uses. As I...

Our EEG group discussion on 'useful' AI tools

2025-03-07T00:00:00Z

Srinivasan Keshav organised this week's EEG group discussion on what AI tools we use for our daily work. I was immediately struck by how few tools there are that are actually making us more productive, so I jotted down notes as the discussion was going on. Personally, the only tool I've found that's (only just recently) making me more productive is agentic coding, which I wrote about a few days ago . Since then, I've been mildly obsessively forking off ideas I've wanted to try for years (like converting RFCs to OCaml code) and greatly enjoying myself. Patrick Ferris and I have been looking for how to do this more ethically, and the best I ran across was the IBM AI ethics guidance and their granite models , but not much else. Any pointers to other models that don't violate open source licensing norms would be...

devaluation of ai

2025-03-06T10:36:00Z

Change for the Machines (with apologies to Pat Cadigan) AI was about models where money equals compute equals big data equals valuations. So all the money going in was to finance compute, thinking where the value lies, and every valuation of every company was about how much compute they had so it was all fake. Companies valuations were just how many H100s they had (compute capacity), as if it correlates to better models (even though they’re usually just wrappers. DeepSeek, and other Chinese models broke that which pissed everyone off, private and public investors included, because it casted doubt on the valuation methodologies, namely that energy, compute, data centres and number of chips were essentially fixed costs and the valuation of companies (and their output) could be measured on...

Are you still using OCaml 4.08 or earlier? If so, we need to know

2025-03-05T00:00:00Z

I started pushing OCaml Docker images over to the Docker Hub in around 2017, to support the burgeoning automated build infrastructure around the use of the language. Back then, OCaml 4.06 was the latest release, and so I wrote an ocaml-version library to track the release metadata. It has been a bit of a success disaster, as that library now tracks every release of OCaml in the modern era, and also backs the automatic building of a huge array of compiler versions and variants across Linux and Windows . The problem is...we're now building the full set of images from OCaml 4.02 onwards through to the latest OCaml 5.3.0 release, which is unsustainable for obvious reasons; despite the hosting being kindly sponsored by Docker , we must also consider the carbon footprint of our infrastructure. So the...

Weeknotes: 3rd March 2025

2025-03-03T09:07:12Z

Last week Using Go with Wasm and Web Workers As indicated last week , I spent some time learning how to use both Go for Web Assembly and with Web Workers for parallelism in the browser. I wrote a long post about the details if that's of any interest. Webplats To make the above post I had to actually do tweaks to Webplats , the software I wrote that hosts this site. The technical details aren't that exciting, but it's just nice to observe how liberating it is to be able to hit a limitation of the hosting tool you're using and be able to quickly change it to remove that limitation. I did however also have to disable code syntax highlighting as hilite (or possibly one the libraries it uses to tokenise the code) is struggling with Go, and no syntax-highlighting seemed a better fallback...

Mon 3 Mar 2025

2025-03-03T00:00:00Z

Previous: Tue 25 Feb 2025 Next: Mon 10 Mar 2025 Babel A Debian encoding in PubGrub DONE provides with virtual packages here DONE cross ecosystem resolutions between opam and debian Done in babel . Some examples follow. conf-gmp #[test] fn test_conf_gmp_variables() -> Result<(), Box> { let root = OpamPackage::Root(vec![ ( OpamPackage::Base("conf-gmp".to_string()), Range::singleton(OpamVersion("4".to_string())), ), ( OpamPackage::Var("os-family".to_string()), Range::singleton(OpamVersion("debian".to_string())), ), ( OpamPackage::Var("os-distribution".to_string()), Range::singleton(OpamVersion("debian".to_string())), ), ]); solve_repo( BabelPackage::Opam(root), BabelVersion::Opam(OpamVersion("".to_string())), "../pubgrub_opam/opam-repository/packages",...

Oh my Claude, we need agentic copilot sandboxing right now

2025-03-02T00:00:00Z

Yaron Minsky nerdsniped me last week into getting OCaml to drive the 80s-retro RGB Matrix displays. I grabbed one from the local Pi Store and soldered it together with help from Michael Dales . But instead of writing OCaml bindings by hand, we thought we'd try out the latest agentic CLI called Claude Code released last week to see if we could entirely autogenerate the bindings. TL;DR: Claude Coder generated working OCaml code almost from scratch, ranging from C bindings to high-level OCaml interface files and even Cmdliner terms, but needs a more sophisticated sandboxing model before something goes horribly wrong. So much potential and so much danger awaits us. Coincidentally Cyrus Omar and Patrick Ferris and I wrote about this a few months ago. Read on... Wiring up the display to my...

A hardware description language using OCaml effects

2025-03-01T00:00:00Z

A hardware description language using OCaml effects This is an idea proposed in 2025 as a Cambridge Computer Science Part III or MPhil project, and is available for being worked on. It may be co-supervised with KC Sivaramakrishnan and Andy Ray . Programming FPGAs using functional programming languages is a very good fit for the problem domain. OCaml has the HardCaml ecosystem to express hardware designs in OCaml, make generic designs using the power of the language, then simulate designs and convert them to Verilog or VHDL. HardCaml is very successfully used in production at places like Jane Street , but needs quite a lot of prerequisite knowledge about the full OCaml language. In particular, it makes very heavy use of the module system in order to build up the circuit description as an OCaml data...

An Introductory Guide on How to Create Beautiful Slides

2025-02-28T17:59:20Z

Frank Feng is a first-year Ph.D. student in the Department of Computer Science and Technology at the University of Cambridge. His research interests lie at the intersection of machine learning and earth sciences, with a particular focus on the application of self-supervised learning in remote sensing.

2025-02-28 Weekly Notes

2025-02-28T00:00:00Z

This week I focused mostly on checking why I had some empty rows in the estimation of the canopy cover and the tree count from the Vegetation Object Model. After investigating the code, I found that the issue was behind a try/except clause that was masking corrupted files. Now, this was the intended behaviour originally, as some files can’t be read, however, because the data processing is done at the geographic statistical level (read ONS ), and the VOM tiles come in 5x5 km files (following GB National Grid), even when one file can’t be read, the other ones wouldn’t be processed either. With the new small correction, only the corrupted files are not processed and the rest of the files are processed as intended. So from the 5402 tiles, only 26 can’t be read. In addition to this, I...

The AIETF arrives, and not a moment too soon

2025-02-28T00:00:00Z

The IETF announced their new AI Preferences Working Group (AIPREF), which will "work on standardizing building blocks that allow for the expression of preferences about how content is collected and processed for Artificial Intelligence models" . This is quite well timed; the IETF tries not to standardise too early before there is running code but also needs to move before it's too late and a bad defacto standard is chosen . The AI world seems to be at that nexus point right about now, with GPT 4.5 seemingly hitting a scaling wall and possibly triggering the start of a renewed data scraping frenzy. How do websites interact with AI crawlers right now? I've found when developing my own website there are a number of approaches to interacting with automated data crawlers. For the record, over 90% of...

Blogging BIOSPACE25!

2025-02-28T00:00:00Z

Blogging BIOSPACE25! 28th February, 2025 Hello world , this is my first Jekyll blog post. With that formality out the way… a couple weeks ago I headed off to the Biospace conference at the ESA-ESRIN Observation Center in Frascati, Italy. While I was only there for 2 days, there was a lot to be excited about. Just past the entrance to the ESA-ESRIN Observation Center, and also past some fairly intense security! My big takeaway from the opening speeches was that this is the first year that the ESA is spending more on building out its data science capabilities than it is on putting satellites into space. To me, this is indicative of the fact that the marginal benefit from putting effort into effectively wrangling huge amounts of data is now greater than that from collecting huge amounts of...

Tue 25 Feb 2025

2025-02-25T00:00:00Z

Previous: Mon 17 Feb 2025 Next: Mon 3 Mar 2025 Babel: opam repository with pubgrub-opam A Debian/Alpine encoding in PubGrub, which I think should be much simpler than Opam DONE Got the basics in pubgrub-debian Here’s a solve of openssh-server: (openssh-server, 1:7.9p1-10+deb10u2) -> (libssl1.1, >=1.1.1), (openssh-client, 1:7.9p1-10+deb10u2), (libc6, >=2.26), (libcom-err2, >=1.43.9), (procps, *), (lsb-base, >=4.1+Debian3), (libaudit1, >=1:2.2.1), (dpkg, >=1.9.0), (ucf, >=0.28), (libselinux1, >=1.32), (libgssapi-krb5-2, >=1.17), (libsystemd0, *), (debconf: >=0.5 | debconf-2.0: *, *), (libpam0g, >=0.99.7.1), (openssh-sftp-server, *), (libpam-runtime, >=0.76-14), (libpam-modules, >=0.72-9), (zlib1g, >=1:1.1.4), (adduser, >=3.9), (libkrb5-3, >=1.13~alpha1+dfsg), (libwrap0, >=7.6-4~) (libssl1.1,...

Using Go with Wasm and Web Workers

2025-02-24T15:08:58Z

These are some notes for myself about trying to use Wasm and Web Workers to achieve some level of parallelisation in the browser. This isn't meant to be a comprehensive tutorial, but there are so many broken tutorials or half bits of documentation out there, I thought I should leave myself a note here. This is just the result of an afternoon of spelunking to try and work out how to do this, and should not be considered comprehensive. Example If you're viewing this page directly (rather than via an RSS reader) and your browser supports Wasm, then below you should see a Mandelbrot fractal render into place above, with different chunks appearing at different points (and on Safari you might see some banding, which is it failing to align the canvas tiles properly rather than being an issue with the...

Weeknotes: 24th February 2025

2025-02-24T09:07:12Z

Last Week LIFE As started last week , I finished the changes to let you run the LIFE pipeline using the bulk data downloads from the IUCN Red List website, which give you a bunch of CSV files of species data and then some shapefiles of range data. Previously I'd been using a PostGIS database with all the data in, but not everyone has one of those to hand, and I want to make it was easy as possible for others to run the pipeline to reproduce our work. After being very impressed last week at the behaviour of DuckDB , particularly in the way it'd let me do spatial queries, I did realise that the performance of ST_UNION_AGG , it's version of PostGIS's ST_UNION was significantly slower than just pulling the different ranges for a species and using shapely to unify them in Python. It's still...

Robotics and Sensing for Sustainable Crop Production

2025-02-21T16:26:06Z

Abstract: Crop farming is essential in our society, providing food, feed, fiber, and fuel. We heavily rely on crop production, but at the same time, we need to reduce the production footprint. We aim to address this key challenge by investigating new solutions to produce crops more sustainably. We study novel technology-driven approaches to move toward sustainable crop production. Agricultural robots offer promising directions to address management challenges in agricultural fields or support plant breeding efforts through large-scale trait acquisition. For that, field robots need the ability to perceive and model their environment, predict possible future developments, and make appropriate decisions in complex and changing situations. This talk will showcase our recent developments in...

2025-02-21 Weekly Notes

2025-02-21T00:00:00Z

So back to the routine after coming from Italy for BIOSPACE25. It was a really great experience in the ESA HQ but I got a cold from Italy 😷. This week started by going to the Social and Spatial Epidemiology Module from the PHS MPhil in the Biomedical Campus. It’s originally desgined to help public health students understand that it is a spatial domain, with all that it means. I really enjoyed the course in that I was exposed to things that I wasn’t aware of like redlining and gerrymandering in the US. Special shoutout to Dr James Lewis , invited speaker to the module, who is an active nephrologist in Atlanta, Georgia. His lecture is one of the most engaging I’ve ever seen here in Cambridge. I really enjoyed how he initiated with the story about his aunt and ended with his daughter. He also...

A trio of papers I read on biodiversity and forests this week

2025-02-20T00:00:00Z

This week I've been reading three really nice pieces of work by my colleagues, in the form of a review paper on biodiversity and AI, a benchmark for 3D forest reconstruction using laser scanners and a mobile app for measuring the width of tree trunks. A real bonanza for forest lovers! Review paper on mapping opportunities for AI in biodiversity A paper on ' Harnessing AI to fill global shortfalls in biodiversity knowledge ' just came out in Nature Biodiversity today (via Oisin Mac Aodha ). They start with the baseline present uses of AI (camera traps, acoustic monitoring and improved data analysis) which are pretty well known to anyone in the field, but then introduce a lovely diagram of future uses of AI for biodiversity which includes: Rapid retrieval of existing information means both looking into existing literature, but also the digitisation of existing museum...

Weeknotes: 17th February 2025

2025-02-17T09:07:12Z

Last Week LIFE For the LIFE extinction-risk metric pipeline , all the species data comes from the IUCN Red List , and this week I began a job I'd been intending to do for a while to make that stage pragmatically accessible to people trying to run our pipeline externally. Originally, as LIFE was being developed, the species data was manually downloaded from the Red List website using their bulk download facility, which gives you a ZIP file of CSVs for the empirical data, and then another set of ZIPs that contain the polygon range data. These had then been run through a mixture of hand processing and Python scripting to filter it down to the appropriate species for the LIFE metric. As we came to publication I wanted to automate this, which I did, but at that stage, because I was working closely with the IUCN on...

Vpnkit, Void Processes, LSP Servers

2025-02-17T00:00:00Z

Previous weeklies used strong emphasis to distinguish sections. This comes from Forester 's philosophy about atomicity of the content in your forest . However, subtrees are supported! I quickly hacked together the ability to use subheadings to indicate subtrees . This is strictly less expressive than the \subtree{} of Forester 's default syntax as we cannot close heading sections in Markdown. This weekly uses subtrees. Vpnkit I spent some time this week trying to upgrade vpnkit to OCaml 5. I was originally working on a paper idea which might need benchmarks, but Anil and I decided we could simply point to the port I did and show how it has simplified much of the code. Void Processes Work continued on implementing (and fully exploring) void processes . A lot of the groundwork already...

Mon 17 Feb 2025

2025-02-17T00:00:00Z

Previous: Mon 10 Feb 2025 Next: Tue 25 Feb 2025 Babel: opam repository with pubgrub-opam DONE tests DONE PubGrub 0.3.0 Uv is using the development branch of PubGrub. There’s a lot of improvements to be had since the last release was 4 years ago. The new version bounds will cleanly express version constraints, e.g. stop converting <= 1.0.0 to < 1.0.0.1 It looks like we might be able to manually add conflicts with add_incompatibility Ah, actually this is an internal thing. Completed here . DONE conjunctions and disjunctions in filtered package formula Take "D" { test & > "2.0.0"} as an example. We encode this as, (filtered-package-formula-variable-version, 1.0.0) -> (D {(test & = >2.0.0)}, *) (D {(test & = >2.0.0)}, false) -> (`test`, false) (D {(test & = >2.0.0)}, true) -> (`test`, true), (A,...

Thoughts on the National Data Library and private research data

2025-02-17T00:00:00Z

Over the past year, Sadiq Jaffer and I have been getting an object lesson in how the modern Internet handles researcher access to data, as we've been downloading tens of millions of research papers towards our Conservation Evidence project. This is legally possible via our institutional subscriptions that give us license to fulltexts, and the incredibly helpful head of electronic services at the University Library who wields encyclopedic knowledge of each of our agreements with the hundreds of publishers out there. My thoughts on this then segwayed into recent conversations I've been having about the emerging National Data Library and also with the UK Wildlife Trusts ... The difficulty of access controlled bulk data downloads In late 2023, once we got past the legal aspects of downloading closed access papers [1] it was still remarkably...

Using Low-cost, Research-led, Decentralised Networks

2025-02-14T16:25:58Z

Full Title: Using Low-cost, Research-led, Decentralised Networks to Increase Access to High Quality Microspatial Data on Building Stocks, and the Built and Natural Infrastructure Abstract: The Colouring Cities Research Programme (CCRP) is overseen by an informal, international academic consortium that uses its decentralized research-led network, to co-create and manage, permanent open data/visualisation platforms across countries. These provide standardised, open microspatial data on the characteristics, performance, and short/long-term dynamics of building stocks, and built and natural infrastructure. They also test feedback loops between live streaming, computational inference, and crowdsourcing approaches to improve coverage and reliability of data, and to support cross...

BIOSPACE25 Blog

2025-02-14T00:00:00Z

Day 2 My presentation (available to watch here ) A couple of people approached me about deploying the Terracorder, particularly an organisation that works in training Indigenous communities in the Amazon and Kenya. (Diana Mastracci) Got asked about the technical components of the terracorder (how far can two devices be from each other?) Several people scanned the QR code, so hopefully, they will contact me. A representative of the European Commission asked about the urban tree map (Have to watch the recording because he ran away after the presentation) Poster Session Onkar did really great in his presentation. There are people who are working on using GeoFM for sustainable goals. There is a trend on using as much RS data as possible for different applications. Michael Munk from DHI...

`nmcli` connection to internal WiFi

2025-02-14T00:00:00Z

Using my fancy (?) new(-ish) Linux laptop running NixOS , I finally had cause to connect to our internal Wi-Fi network. This was not entirely trivial due to the various configuration options required. So here goes, for the record, what I did as an aide memoir for me and in case it’s useful for anyone else… First, create the connection – the Wi-Fi network in question is named Internal-CL : $ sudo nmcli connection add type wifi con-name Internal-CL ssid Internal-CL Connection ' Internal-CL ' (8f1ddcc9-4b1f-4e5d-9992-522714685eb4 ) successfully added. Then, configure it: $ sudo nmcli connection edit Internal-CL = == | nmcli interactive connection editor | = == Editing existing ' 802-11-wireless ' connection: ' Internal-CL ' Type ' help ' or ' ? ' for available commands. Type ' print '...

Updated preprint on quantifying biodiversity cost of food consumption

2025-02-12T00:00:00Z

We've uploaded a revised preprint on our ongoing work on quantifying the biodiversity cost of global food consumption , lead by Thomas Ball . This is based on the recently published LIFE metric, combined with supply chain data and provenance modeling. [… 196 words ] Thomas Ball , Michael Dales , Alison Eyres , Jonathan Green , Anil Madhavapeddy , David Williams and Andrew Balmford . Working paper at Cambridge Open Engage . URL (cambridge.org) DOI BIB PDF

Weeknotes: 10th February 2025

2025-02-10T09:07:12Z

Last week Yirgacheffe I hand't planned this, but I spent a day reworking the Yigacheffe APIs towards what I've always thought of as the next major version interface, and had filed under "mañana" due to other work being more important. But inspired by one of the part II projects I'm supervising (final year undergrad for those not from Cambridge), I decided I really needed to fix up the APIs to make them cleaner. Yirgacheffe has some clunky parts to the API where it fails to meet the aim I have that working on maps should result in Python code that clearly follows the methodology rather than dealing with house keeping around the data. In a lot of areas I like to think I've been successful; for example, it will deal with loading in data in small chunks to as to fit within memory. But other...

Teaching, teaching, teaching...

2025-02-10T00:00:00Z

On paper, I don't have that many students. I teach four undergraduates (first year students at Pembroke College) Discrete Maths . I supervise three third year students for their final year project and another one I co-supervise with Michael Dales . However, I do end up spending at least two full days a week on teaching. Something I really enjoy and take seriously. The time it takes is also quite unpredictable; last week for instance all the third year students had their mid-project demonstrations (a five-minute presentation in front of their peers and a few professor-types). My first year students also found two slides particularly challenging to understand from their lectures and asked if I could help explain what was going on, so I produced some materials for that . OCaml Within the OCaml universe I spent a good bit of...

Mon 10 Feb 2025

2025-02-10T00:00:00Z

Previous: Mon 3 Feb 2025 Next: Mon 17 Feb 2025 Babel: solve the opam repository with pubgrub-opam While converting the whole repository I fixed a bug in a package definition DONE on-demand dependency provider opam-repository has a lot of packages, so we only load packages we need TODO add a caching dependency provider with interior mutation DONE parse filtered package formula While the grammar allows for it I can’t actually find any booleans or integers in the dependency formula of Opam. Why are they there? I’m just ignoring them for now so I don’t need to complement logic for comparing them etc. DONE encode filtered package formula in pubgrub Take (A, 1) depending on B {build} as a minimal example. We encode this in the ‘package calculus’ as (A, 1) -> (B {build}, *) (B {build}, lhs) -> (B, *), (`build`, false) (B {build}, rhs) -> (`build`, true)...

The Cambridge "Green Blue" competition to reduce emissions

2025-02-10T00:00:00Z

Carl Edward Rasmussen recently gave a great talk in our group about his thoughts on mechanisms against climate change . He persuasively argued that the Paris Agreement was doing more harm than good by giving the illusion of being a concrete agreement, but is in reality a huge distraction. Our actual emissions have increased since the Paris agreement was signed! Carl argues that a climate system ultimately only responds to collective actions, and without a global cooperative incentive each nation will spring back to their own isolated short-term incentives that lead to an increase in fossil fuel burning. He has just published the " Themis Mechanism " as a simple alternative for equitable global emission reduction ( long form ). This got me brainstorming with Carl about how to test his theories out and we came up with an...

Using AT Proto for more than just Bluesky posts

2025-02-09T00:00:00Z

While Bluesky is taking off like a rocket, a number of us moving towards self sovereign digital infrastructure have been looking at how to use the Bluesky network for other uses than just short-form notes. This is possible because of my colleague Martin Kleppmann 's hard work on the " AT Protocol " that underpins the Bluesky network. Martin recently gave us a deep-dive into the AT proto in the Cambridge security group , which made me look into other uses of it more closely. As background, you may wish to read his paper on the subject which explains the technical architecture extremely clearly. __LINK_8__ One of the key problems this solves is one I'm having with using my ActivityPub -based services at the moment. Each of these services (like my video or microblog sites) do not share a common...

Sway

2025-02-07T00:00:00Z

Published Fri 7 Feb 2025. Tags: technology . I use the sway Wayland compositor, which is a clone of the X11 i3 window manager for Wayland . Wayland is a communications protocol for next-generation display servers used in Unix-like systems which has been adopted as the default display server by Linux distributions including Fedora with KDE, and Ubuntu and Debian with GNOME. I use the sway Wayland compositor. It aims to replace the venerable X display server with a modern alternative. X leaves logic such as window management to application software, which has allowed the proliferation of different approaches. Wayland, however, centralises this logic in the ‘compositor’, which assumes both display server and window manager roles. Status In lieu of a status bar I invoke a...

Entering the Matrix with Hookshot

2025-02-07T00:00:00Z

We've been happy users of Matrix for our group communications in the EEG . Today we've been bringing in more members of the wider group to using it instead of Slack. As part of that, I've set up a cool bot called Hookshot which allows Matrix to be connected to external services such as GitHub and Atom/RSS feeds. This is a test post to demonstrate to the members of the EEG how Matrix and Atom work! The basic idea behind Hookshot is to provide a bridging service to communications rooms hosted on Matrix, in such a way that it can exert administrative control over a room to intercept requests for services (such as adding an Atom feed). The setup for Hookshot can be a little involved as there are lots of encryption keys flying around. In a nutshell, I have a Docker container...

Programming FPGAs using OCaml

2025-02-07T00:00:00Z

With the vast amount of data we have these days for our planetary computing processing, it's naturally tempting to use more hardware offload. The obvious choice, GPGPUs, are not a great fit for the problem due to the difficulty of unlocking high data parallelism for geospatial data. So it's back to an old technology I worked on twelve years ago in the form of FPGAs ! FPGAs are a very flexible way to execute boolean combinatorial logic, but are notoriously difficult to program. We have two possible angles to explore to address this. One is to design more declarative DSLs for data processing that compile to the FPGAs, such as Michael Dales work on Yirgacheffe or Omar Tanner 's work on in-memory compressive computation . The other angle is to work on the low-level API to programming the FPGAs, to get away from...

`just` OCaml

2025-02-07T00:00:00Z

In similar vein to a recent post , I have also started using just when I periodically need to rebuild my OCaml tool 1 ocal . So I ended up replacing the old Makefile with a shiny new Justfile . As it also proved useful in another (more esoteric) tool I wrote for parsing out exam results for my students so I can paste into email easily , I thought I’d put it here for the record. So here it is… 1 Largely due to NixOS upgrades moving tools into different locations. Usual preamble of course: _default : @ just --list Then set some common variables: PWD := env ( " PWD " ) DOCDIR := " _build/default/_doc/_html " BUILDDIR := " _build/install/default/bin " Then set the target — the tool name, in this case ocal (so named as this is an OCaml re-implementation of a tool approximating the trad Unix cal tool): TARGET := " ocal...

Reverse `find`

2025-02-05T00:00:00Z

In the last few days I discovered I needed to search back up the filesystem from $CWD to find the first occurence of a file (specifically, a Justfile but that’s by-the-by). Got bored of doing it by hand so wrote a bash shell function; here ’tis: rf ( ) { local D while ! eza - l " $ { D := . } / $ 1 " ; do # first, check `$CWD` [ " $ ( realpath " $ D / $ 1 " ) " == " / $ 1 " ] && break # stop if we hit `/` already D = $ D /.. # else, iterate one layer up done } Invoke as (e.g.,) rf Justfile . Alternatively, as a one-liner: F = Justfile ; while ! eza - l $ { D := . } / $ F ; do [ " $ ( realpath $ D / $ F ) " == " / $ F " ] && break ; D = $ D /.. ; done ; unset D

Fake papers abound in the literature

2025-02-04T00:00:00Z

Sadiq Jaffer sent along this piece in The Conversation last week about the remarkable number of academic papers that are now AI generated. The numbers of these papers are probably underestimated: These papers are absorbed into the worldwide library of research faster than they can be weeded out. About 119,000 scholarly journal articles and conference papers are published globally every week, or more than 6 million a year. Publishers estimate that, at most journals, about 2% of the papers submitted – but not necessarily published – are likely fake, although this number can be much higher at some publications. -- Frederik Joelving et al, The Conversation What caught my eye in this article is their development of the Problematic Paper Screener , which the good folks at Retraction Watch developed. It works with high precision to detect...

Weeknotes: 3rd February 2025

2025-02-03T09:07:12Z

Last week STAR I finally got time to sit down and catch up with trying to recreate STAR , the IUCN's Species Threat Abatement and Restoration metric, which is similar to, but different from, the LIFE metric that I work on. Before Christmas I'd been working with Chess Ridley who maintains and runs the current STAR pipeline for the IUCN, trying to match up my species selection process with hers. Since then I'd found out some more nuances to both the IUCN redlist data and to my ability to write SQL that queries JSON fields in Postgres, and I'm now sitting closer to Chess's results, but not quite there yet: AMPHIBIA Total Chess: 3249 Total Michael: 3260 In Chess data, not in Michael data: 0 In Michael data, not in Chess data: 11 AVES Total Chess: 2180 Total Michael: 2214 In Chess data,...

Keyboard

2025-02-03T00:00:00Z

Published Mon 3 Feb 2025. Tags: technology . I developed RSI symptom’s at the tail of writing my MPhil dissertation and started looking for ways to work more ergonomically. As someone who’s often typing in one form or anther 8+ hours a day one obvious aspect to look at was my keyboard. I’m using the ZSA Moonlander as my daily driver which I purchased with my PhD stipend machine budget. It’s an ergonomic ortholinear split mechanical keyboard. The ortholinear layout has allowed me to learn how to touch type (I could just never get the hang of it on a normal keyboard). Using keyboard-centric editors like Vim and Emacs , as well as Firefox Tridactyl , has helped my reduce mouse usage which had also proved aggravating. I was briefly looking at 3D-printing a custom keyboard, like the Dactyl-ManuForm , but...

Firefox

2025-02-03T00:00:00Z

Published Mon 3 Feb 2025. I use Firefox as my default browser. You can see my configuration in Nix OS here . Having it in Nix is handy so I don’t have to sign into a Firefox account on a live ISO or new machine install in order to get the environment I’m used to. Tridactyl I use Tridactyl for many useful Vim-like bindings in Firefox. Similar to Firefox you can see my configuration here . Some useful bindings are: h j k l scroll f enter ‘hint’ mode, using keystrokes to follow a link F opens a link in a new tab ;w open in a new window ;y copy link v highlights an element of text once here we can use caret mode to select text yd duplicate tab My custom binding i let’s me copy a link to a URL fragment with hint mode; bind i hint -JFc [id] e => {...

Mon 3 Feb 2025

2025-02-03T00:00:00Z

Previous: Mon 27 Jan 2025 Next: Mon 10 Feb 2025 Babel I’ve been focusing on our new package manager to rule them all, code name `Babel’, this week. PubGrub opam dependency provider I’m making progress so supporting Opam ’s full feature set with the Rust implementation of the PubGrub version solving algorithm, which we’re aiming to use for a cross-ecosystem solver. Previous work has used the Opam semantics for multiple ecosystems. The PubGrub encoding is quite similar to the `HyperRes’ formalism we defined for a minimal subset of functionality between ecosystems.. DONE I’ve added support for the complicated ordering of opam versions - here DONE I’ve added an encoding of boolean logic in Opam’s package formula - here We support version formula with PubGrub’s Range operations (e.g. union, intersection, etc) We support...

Satellites are getting too good for forest carbon?

2025-02-03T00:00:00Z

There's a letter in Science today from a bunch of well known remote sensing researchers that make the unusual point that modern satellite resolution is getting too good to be accurate for forest carbon estimation. Many new satellites can resolve fine features on the landscape, and even some individual trees outside forests, but this resolution (0.3-5m) is too high for mapping forest carbon. Forest carbon has a natural resolution constraint: the size of an individual tree. To create these maps, tree data from the ground are required because there is no direct measure of tree carbon nor any way to accurately divide trees into smaller components from space. [...] Because most carbon in a forest is stored in large trees, map resolutions should at minimum exceed the crown diameter of a typical large...

Emacs

2025-02-02T00:00:00Z

Published Sun 2 Feb 2025. Last update Tue 18 Feb 2025. Tags: technology . I started using Emacs for the mu4e mail client after using and liking the mu mail indexer on the command line (read more about my mail client setup ). Similarly, I was using the ledger-cli accounting system, and found the Emacs minor mode to be great. Then I got curious about Emacs Org Mode and started using it for note taking and task management. Now I’m using Emacs as my primary development environment. I realised that configuring Emacs to my liking would be like second job, which I didn’t have the bandwidth for, so I started using the ‘distribution’ Doom Emacs . It enables Vim bindings everywhere with Evil Mode which, coming from Vim , made it much easier to get up to speed. The Emacs philosophy...

Disentangling carbon credits and offsets with contributions

2025-02-02T00:00:00Z

The terms carbon credits and carbon offsets are often used interchangeably, but are in fact two distinct concepts. I've spent a nice Sunday morning reading up on some recent articles that Bhaskar Vira sent me which introduce a third term, known as "carbon contributions" . Rather than this adding confusion, I found it helped me clarify my own thoughts on the matter, which I note down here in draft form. (Update 7th Feb: I've revised this several times after many discussions this week, especially with David A Coomes and Srinivasan Keshav , with full list of credits in the end) What are carbon credits and offsets? A carbon credit aims to quantify the net climate benefit resulting an intervention that alters some CO2 emissions that would otherwise have gone into the atmosphere in a business-as-usual counterfactual scenario. While...

(Neo)Vim

2025-02-01T00:00:00Z

Published Sat 1 Feb 2025. Last update Tue 18 Feb 2025. Tags: technology . I used to use Neovim as my main development environment. I like the fact that it’s small and fast, and that I can use it over an SSH connection. Neovim is a fork of Vim that I use for the Lua configuration and built-in Language Server Protocol support. However, I mainly use Emacs these days. I extensively use evil (a Vi(m) compatibility layer for Emacs), so I haven’t really stopped using Vim but instead just use it inside a big Lisp machine. You can find my configuration in my Nix OS repository here . Below are my working notes on learning and using (Neo)Vim. References: :help usr_toc.txt :help index ZZ / ZQ exit with / without saving basic motions h j k l i I a A 0 $ w e b W E B...

Using computational SSDs for vector databases

2025-02-01T00:00:00Z

Using computational SSDs for vector databases This is an idea proposed in 2025 as a Cambridge Computer Science Part III or MPhil project, and is available for being worked on. It may be co-supervised with Sadiq Jaffer . Large pre-trained models can be used to embed media/documents into concise vector representations with the property that vectors that are "close" to each other are semantically related. ANN (Approximate Nearest Neighbour) search on these embeddings is used heavily already in RAG systems for LLMs or search-by-example for satellite imagery. Right now, most ANN databases almost exclusively use memory-resident indexes to accelerate this searching. This is a showstopper for larger datasets, such as the terabytes of PDFs we have for our big evidence synthesis project, each of which...

Deepdive into Deepseek advances

2025-02-01T00:00:00Z

Srinivasan Keshav posted a link to this excellent deepdive by Prasad Raje of Udemy into the advances that DeepSeek R1 has made from a perspective of the core technology. Multi-headed Latent Attention (MLA). In the famous Google " Attention is all you need " paper, the attention block is responsible for a lot of the magic of LLMs but is also compute heavy [...] Deepseek has innovated here with Multi-headed latent attention - which essentially reduces the size of matrix multiplication applied to generate the K,V vectors that are inputs into the attention block. Combined with KV Caching, this reduces the memory needs [...] Mixture of Experts (MoE). The key idea here is that instead of feeding each token through one massive FFN , break down the single FFN into a number of smaller FFNs and route each token...

Affordable digitisation of insect collections using photogrammetry

2025-02-01T00:00:00Z

Affordable digitisation of insect collections using photogrammetry This is an idea proposed in 2025 as a Cambridge Computer Science Part III or MPhil project, and is available for being worked on. It may be co-supervised with Tiffany Ki and Edgar Turner . Insects dominate animal biodiversity and are sometimes called " the little things that run the world ". They play a disproportionate role in ecosystem functioning, are highly sensitive to environmental change and often considered to be early indicators of responses in other taxa. There is widespread concern about global insect declines [1] yet the evidence behind such declines is highly biassed towards the Global North and much is drawn from short-term biodiversity datasets [2] [3] . The Insect Collection at the University Museum of Zoology, Cambridge...

How to publish custom Homebrew taps for OCaml

2025-01-31T00:00:00Z

Now that I've switched to a new website, I'm working on open-sourcing its components. I've got a lot of small OCaml scripts that are all work-in-progress, and so not quite suitable to be published to the central opam-repository but I still need be able to run them conveniently on my own self-hosted infrastructure. I mainly use a variety of macOS and Linux hosts [1] and I want a workflow as simple as " brew install avsm/ocaml/srcsetter " and have it install a working binary version of my CLI utility. In this case, it's srcsetter , a simple tool I knocked up to generate the responsive images on this website. Luckily, Homebrew has made this really easy for us! They have a BrewTestBot that integrates with GitHub Actions to automate the compilation of binary packages for us, all from a convenient PR-like...

PhD viva, UK style

2025-01-31T00:00:00Z

Having recently, happily, had several PhD students completing in short order, I’ve been approaching external PhD examiners. Occasionally I find myself asking someone who’s not done any / many in the UK previously. As our system – as all such systems! – is a bit different to those in other parts of the world, I’ve written a few notes on a couple of occasions about what to expect. So I figured I might as well publish them. What follows is my impression / understanding based on experience here in the Department of Computer Science & Technology , Cambridge University ca. 2025. Your Mileage May Vary anywhen and anywhere else, including anywhere else in the UK – check local regulations to be sure. In terms of process, the system here is that candidates submit their complete dissertation and then undergo a “viva voce” (oral examination)....

Position paper on scientifically credible carbon credits

2025-01-30T00:00:00Z

My colleagues Thomas Swinfield and Eleanor Toye Scott lead the publication of a comprehensive report of the steps the voluntary carbon market needs to take to restore its scientific credibility, with input from many of us in 4C and beyond. establishing common standards for carbon quantification and accounting, to cover additionality, leakage and permanence. avoiding perverse incentives and align the motivations of all stakeholders with high-integrity outcomes. [...] issuing all carbon credits based on trusted primary observations. making all the data needed to reproduce carbon calculations available in standard file formats. [...] reporting social and biodiversity dimensions of projects separately from carbon calculations. integrating DMRV methods into carbon and biodiversity accounting...

Arise Bushel, my sixth generation oxidised website

2025-01-29T00:00:00Z

This website has been through quite a few iterations over the years. The first version in 1998 was written in Perl and hosted on OpenBSD ; the second was rewritten in 2000 when I got commit access to PHP ; the third rewrite became a hybrid OCaml/PHP/Perl special in 2004 in Blosxom ; then the forth rewrite around 2013 got turned into a unikernel in MirageOS; then the fifth in 2019 then transitioned to an OCaml static site generator hosted on a prerelease multicore OCaml webserver . So the sixth generation now needs something to continue the grand Rube Goldberg tradition of helping me learn the latest and greatest in systems technology. And so here it is! The site is now written in a bleeding-edge unreleased variant of OCaml with extensions based around Rust-like type system features activated, including rather exciting...

AT Protocol and an IR for Wikis

2025-01-27T00:00:00Z

AT Protocol This week I've been diving into the AT Protocol . The Authenticated Transfer Protocol, aka atproto, is a decentralized protocol for large-scale social web applications. The protocol could be a candidate for the glue that holds together a distributed, computational wiki network. The protocol, it seems, is very similar to IPFS . Thankfully, a few years ago, I was working on building out a suite of OCaml libraries for working with IPFS . For example, ocaml-cid , self-describing content-addressed identifiers. # let s = " zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA " ; ; val s : string = " zb2rhe5P4gXftAwvA4eXQ5HJwsER2owDyS9sKaQRRVQPn93bA " # let cid = Cid . of_string s |> Result . get_ok ; ; val cid : Cid . t = < abstr > # Cid . pp_human Format . std_formatter...

Mon 27 Jan 2025

2025-01-27T00:00:00Z

Previous: Mon 20 Jan 2025 Next: Mon 3 Feb 2025 babel DONE multi-version optional dependencies for PubGrub Which as far as I can tell hasn’t been done before. PubGrub opam dependency provider DONE json representation of opam repositories using opam2json DONE parse json representation to rust using serde TODO minimal example DONE opam_version.rs done and MVP with integer versions, but the full ordering is complicated DONE index.rs DONE opam_deps.rs DONE main.rs TODO proper opam version parsing (a la Debian) https://opam.ocaml.org/doc/Manual.html#Version-ordering TODO conflicts TODO boolean logic (package formula) TODO features (optional dependencies) TODO parse repo on the fly in the dependency provider TODO create babel-repository and babel-solver TODO explore error reporting in these exotic encodings what is the research advance here? tool to solve across...

Fontsizing in LaTeX

2025-01-24T00:00:00Z

I recently had colleagues hit an issue that I have hit myself in the past, and so I finally decided to figure out a fix. Specifically, when building EPSRC research proposals in LaTeX, getting a complaint that the font size is non compliant – it should be 11pt Arial, but the standard LaTeX options generate something slightly smaller, with Adobe Acrobat and Microsoft tools both reporting a size of 10.45pt or so. One proposed solution was the to add the following in the preamble: \ usepackage { anyfontsize } \ AtBeginDocument { \ fontsize { 11bp } { 13.35bp } \ selectfont } …but that did not work unfortunately. After some poking about and staring at output and searching the interwebs, it appears that this was triggered, at least for me, by the fontspec package that was being used to sort...

Updated preprint on LLMs for evidence-based decision support

2025-01-23T00:00:00Z

We have just updated our preprint on using LLMs for evidence decision support with more evaluation results and corrections from peer review. Our findings suggest that, with careful domain-specific design, LLMs could potentially be powerful tools for enabling expert-level use of evidence syntheses and databases. However, general LLMs used "out-of-the-box" are likely to perform poorly and misinform decision-makers. By establishing that LLMs exhibit comparable performance with human synthesis experts on providing restricted responses to queries of evidence syntheses and databases, future work can build on our approach to quantify LLM performance in providing open-ended responses. See also the fantastic EEG seminar talk that the student group who worked on this over the summer gave towards the...

spindizzy rabits in space - what to do with the old cavendish portakabins...

2025-01-21T13:10:00Z

now the cambridge physics dept have their Dyson Sphere ready to move into (kind of, ok so it isnt an actual sphere) the old building should be ready for its next role - this should be as a vehicle to get the rabbits (who have moved from their burrows around the ponds into the buildings vacated) to a new home around a friendly exoplanet - I am sure some of the astronomers could have with the celestial navigation.. spindizzy engines are two a penny. -we just need to vaccum proof the buildings - a supply of saran wrap and gaffer tape will do.

Forester, ICFP, Wikis

2025-01-20T00:00:00Z

This week was my first full week back from the break and I found it challenging trying to get back into what I had been working on previously. ICFP Papers In conversation with Anil , we looked at options for submitting a paper to ICFP. I wrote up some notes on some options we discussed . Forester I spent some time this week converting this website to using Forester . I'm not a huge fan of the syntax, especially as a lot of my site's content was already in markdown. So I wrote a markdown frontend to forester which is available on Github . The markdown frontend integrates very nicely and only a few changes were needed in the core logic of Forester itself. Additionally, for any features not directly supported in markdown there is an escape hatch using code blocks such as: ```forester...

Mon 20 Jan 2025

2025-01-20T00:00:00Z

Previous: Mon 13 Jan 2025 Next: Mon 27 Jan 2025 babel DONE PubGrub multi-version optional dependencies TODO reread file:2024-splash-hyperres-review.md TODO spatial DSL paragraphs for Roy met with Roy and gave him the gist bidirectional lens uv PubGrub index uv toolchains - special cross compiling get rid of conflicts name fusing - 1 to n home url metadata source archive many to many hashes download them all – find a backdoor explore and splunk keep eyes open demons lurking in this Spatial Name System Sensor Networks TODO try capnproto esp32 if that doesn’t work try ocaml-esp32 on ocaml 4 if that doesn’t work try ocaml 5 for risc-v Spatial DLS ICFP functional perl TODO read [1] https://en.wikipedia.org/wiki/Functional_reactive_programming https://en.wikipedia.org/wiki/Reactive_programming TODO create a spatial DSL afternoon write out a DSL cotrol lights DSL capnroto vs DNS dsl walk lights effects capnproto...

`just` LaTeX it!

2025-01-18T00:00:00Z

I have recently become a fan of just as a replacement for the venerable make . I find that nowadays I rarely need the built-in dependency rules that make provides. Perhaps more radically, I also rarely need to write my own as the prevelance of format-specific build tools such as cargo , uv , latexmk and the like mean I don’t need to write my own either. Recently, while writing references and then helping out get submissions for Rossfest consistently formatted and building cleanly, I found myself extending my various just targets for LaTeX . So I thought I’d document them here. I always begin my Justfile with the apparently idiomatic “just show me the targets and associated help text” target: _default : @ just --list This seems considerably...

Mon 13 Jan 2025

2025-01-13T00:00:00Z

Previous: Mon 6 Jan 2025 Next: Mon 20 Jan 2025 babel TODO opam-repo maps [2/3] DONE https://github.com/mtelvers/pacman2opam/ it seems like firefox doesn’t install _build/default/bin/main.exe --repo /dev/shm/packages pacman-firefox [ERROR] No solution Can't find all required versions. Selected: pacman-acl.2.3.2-1 pacman-adobe-source-code-pro-fonts.2.042u+1.062i+1.026vf-2 pacman-adwaita-cursors.47.0-1 pacman-adwaita-icon-theme.47.0-1 pacman-adwaita-icon-theme-legacy.46.2-3 pacman-alsa-lib.1.2.13-1 pacman-alsa-topology-conf.1.2.5.1-4 pacman-alsa-ucm-conf.1.2.13-2 pacman-aom.3.11.0-1 pacman-at-spi2-core.2.54.1-1 pacman-attr.2.5.2-1 pacman-audit.4.0.2-3 pacman-avahi.1_0.8+r194+g3f79789-3 pacman-bash.5.2.037-1 pacman-brotli.1.1.0-3 pacman-bzip2.1.0.8-6 pacman-ca-certificates.20240618-1 pacman-ca-certificates-mozilla.3.107-1...

LIFE metric published in Royal Society Phil Trans B

2025-01-09T00:00:00Z

After some years of hard work, our Mapping LIFE on Earth biodiversity metric was published today in a special issue of the Royal Society Philosophical Transactions B! The idea behind LIFE is that although human-driven habitat loss is known to be the greatest cause of the biodiversity crisis , we do not yet have robust spatially explicit metrics that quantify the relative impacts of human actions on species extinctions. And that's what LIFE provides: a way to compare the relative impacts of some landuse anywhere in the world, in a manner that is globally applicable. [… 409 words ] Alison Eyres , Thomas Ball , Michael Dales , Thomas Swinfield , Andy Arnell , Daniele Baisero , América Paz Durán , Jonathan Green , Rhys Green , Anil Madhavapeddy and Andrew Balmford . Journal paper in Philosophical Transactions of the Royal Society (vol 380 issue 1917). URL...

Mon 6 Jan 2025

2025-01-06T00:00:00Z

Previous: Mon 30 Dec 2024 Next: Mon 13 Jan 2025 STRT read PubGrub docs https://github.com/dart-lang/pub/blob/master/doc/solver.md is this actually being used for cargo? feature resolution ah, uv TODO read [1] [1] C. Cabrera, A. Paleyes, and N. D. Lawrence, “Self-sustaining Software Systems ( S4 ): Towards Improved Interpretability and Adaptation ,” Jan. 21, 2024. [Online]. Available: http://arxiv.org/abs/2401.11370 . [Accessed: Jan. 15, 2025]

Introducing `bibtox`

2025-01-03T00:00:00Z

After some time using various tools and scripts to format and sort my files of BibTeX/BibLaTeX entries, I finally gave up back in March and wrote one myself – bibtox . This replaced some very nasty combination of server-side Python and in-page CoffeeScript , plus a third-party tool bib2bib I would run on an ad hoc basis. Per the README , this processes either a stream of entries on stdin or a set of files arranged into sections specified by a simple configuration file. It outputs sorted or unsorted, either as canonicalised entries or marked up HTML. And it means that, hopefully, finally, my list of publications on this site is consistently formatted and sorted. Completeness is, of course, a process rather than a state so achieving that is left for other times and places. Issues or...

computer science is quite a young subject...

2024-12-31T07:09:00Z

...but not as young as it was - I joined ucl. for the last 2 decades of the previous century (and millenium) and while I was there, no-one passed away. the department had only just crystallised out of the statistics and computing department (the stats department being I think one of the first in the world). 20 years after I left, my PhD advisor Peter Kirstein passed away. Until then, I think, most the people that had worked in UCL CS were still around. However, I joined the Computer Lab, a 75 year old department (now with a less unique name), that came out of mathematics, in Cambridge in 2001, and have been there since. But during that time, rather a lot of colleagues there have passed, including (in no particular order), Ross Anderson , John Daugman , Maurice Wilkes, Robin Milner , Mike Gordon, Karen Spärck Jones...

Mon 30 Dec 2024

2024-12-30T00:00:00Z

Previous: Mon 16 Dec 2024 Next: Mon 6 Jan 2025 TODO Spatial Name System camera traps on 4g network no local connectivity what if we could resolve our location to a domain which we could resolve to link local addresses? naming discoverability

the north london book of the not quite dead

2024-12-21T11:52:00Z

The best kept secret at "5" was how you got a new identity. the north london book of the not quite dead was a list of reliable people who had accumulated decades of mathem and could provide a subset of it to anyone needing a new name/face, collection of postcards, convincing travel brochures, expired passports, driving licenses, theater and museum club membership etc etc Json Bourne was in need of a new skin - his command line was completely exposed, and if he wanted to retain one iota of quantum supremacy, he needed to stop the supervillain known simply as "The Cloud"without giving away his whole herd of mastodon instances. Json checked into the Assembly House where he new he could meet Gold or possibly if he was really lucky, Robert Smith, who could tell a tale or two, and certainly...

Retreat from TUM

2024-12-18T00:00:00Z

Ok, ok, I exaggerate – it’s not really that far. But any time I have to set the alarm for 02.30, it feels like it’s a long long way away! TU Munchen host an annual retreat , and thanks to Prof. Joerg Ott I was invited this year for the first time. It’s held in TUM’s Retreat Centre at Raitenhaslach , originally a Cistercian monastery which is a lovely location except that it’s a bus ride from the hotel used and a bus and three trains from the nearest airport – which is, mildly confusingly, Salzburg not Munich . The latter point made me assume that there would be good directions via public transport from Salzburg, but that turned out not to be the case. And as Google Maps is, at best, patchy in terms of public transport coverage in this part of the world – it doesn’t know about all the buses at least – I thought...

Mon 16 Dec 2024

2024-12-16T00:00:00Z

Previous: Mon 2 Dec 2024 Next: Mon 30 Dec 2024 babel opam-repo maps TODO hyperres needs a file format TODO get a JSON format, including version formula opam formulas variables as packages boolean algebra represeted with disjuntions etc I’m wrestling with the problem of expressivity of package managers. What is representable in a SAT problem is consistent across package managers, but there are varying degrees of pre-processing in DSL in different ecosystems. For example, Debian has a pretty simple language for expressing dependency constraints, but Nix has the Nix DSL, opam has filters and variables. These are basically a way of expressing the constraint language with various degrees of pre-processing. It’s fine to unify the decision problem representations, but how can we unify the different...

Optimising Sustainable Energy with Functional Programming

2024-12-13T18:23:12Z

Abstract: This talk describes some results from a collaboration between Computer Science, Physics, and Climate Impact Research on theories and tools for performance optimisation of strongly coupled physical systems with a large parameter space. The first part of the talk discusses computing optimal policies; we have used these techniques for climate decisions and for fusion energy designs. The second part of the talk will focus on one particularly important concept: the Pareto-front, which mathematically captures the trade-offs between two (or more) conflicting objectives. The core object of study is an expensive black-box function computing multiple objectives, for which we approximate the Pareto front using adaptive mesh refinement. Bio: Patrik Jansson is a professor in the Computer...

The Partially Dynamic Web

2024-12-08T11:59:48Z

Background I have three websites (this one, my personal site , and one for my luthiery endeavours ), and despite each starting out with a different technology stack, for the last few years I’d migrated them all to the Hugo static site generator , as a way of making it easier for me to mess around with. Without a fixed database, I could more easily readily structure the content as I wanted it, I had more freedom over templating, and ultimately it’s less resource intensive to compile the site occasionally and just serve static files than keep dynamic infrastructure running for what is a set of low traffic websites. At least in theory, we’ll come back to this last point. Like most static site generators, Hugo uses a system called Front matter , where you store each page as a markdown file of content, and some YAML at the...

Smart Grid Trust Assessment

2024-12-06T16:40:43Z

Abstract: Energy systems are highly complex. State determination and detection of anomalies, faults or even attacks are only possible to a limited extent with traditional approaches. This talk will investigate how such systems can be planned and operated in the future in the area of conflict between high automation and trust by human operators. Bio: Sebastian Lehnhoff is a Full Professor of Energy Informatics at the University of Oldenburg. He received his doctorate at the TU Dortmund University in 2009. Prof. Lehnhoff is chairman of the board of the OFFIS Institute for Information Technology and speaker of its Energy R&D division. He is a board member of the section „Energy Informatics“ within the German Informatics Society (GI) as well as an active member of numerous committees and...

Horizon scan on AI and conservation published

2024-12-05T00:00:00Z

Back in July 2024, a large group of conservation and computer scientists got together in the CCI to prioritise the storm of AI-related projects that have been kicking off around the world. Our key goal was to harness AI to accelerate the positive impact of conservation efforts, while minimising harm caused through either the direct or indirect use of AI technologies. The first horizon scan resulting from this has just been published in Trends in Ecology and Evolution. If you're looking for a gentle introduction to some of the terms in AI from a non-experts perspective, the first section does a good job of defining a glossary as well. [… 118 words ] Sam Reynolds , Sara Beery , Neil Burgess , Mark Burgman , Stuart Butchart , Steven J. Cooke , David A Coomes , Finn Danielsen , Enrico Di Minin , América Paz Durán ,...

Mon 2 Dec 2024

2024-12-02T00:00:00Z

Previous: Mon 18 Nov 2024 Next: Mon 16 Dec 2024 got dodo working again https://github.com/RyanGibb/eon/commit/02176dcb6bd05ae13c0e4187c0dd8873ceff979f but reflecting on it, I don’t think it’s a good use of time eon capability interface added a shell capability to eon https://github.com/RyanGibb/eon/commit/f6c3b13855b0ff9e09b96ccfd2b0b27591390f14#diff-b498faffd1d896b75bce4fc925bfaa48934b8cdb9de9f52d9e8d276ab9161fca now if we could add stun/turn etc…

Towards verifiable, privacy-preserving carbon emissions claims

2024-12-01T00:00:00Z

Customers of online services may want to take carbon emissions into account when deciding which service to use, but it's currently difficult to do so due to the lack of reliable emissions data that is comparable across online services. There's a lot of muddled data out there, and calculating accurate carbon emissions across a computing pipeline involves a number of stakeholders, none of whom are incentivised to accurately report their emissions for competitive reasons! In this LOCO paper, Jessica Man lead our exploration of mechanisms to support verifiable and privacy-preserving emissions reporting across a chain of energy suppliers, cloud data centres, virtual machine hosting services providers and cloud services providers. The idea is that all of this can ultimately be exposed to...

Towards a frugal userspace for Linux

2024-12-01T00:00:00Z

All the work we've been doing on biodiversity (such as LIFE ) comes at a fairly large computation and storage cost due to the amount of data that we churn through. This gets worse when you consider the exploratory nature of science -- we sometimes just need to mess around with the large dataset to test hypotheses which are often shown to be wrong. So then, when the LOCO conference came around, we wrote up our thoughts on what a frugal Linux userspace might look like. The key insight is that the Linux kernel already exposes a number of namespace mechanisms (that we use in Docker, for example), and so we explore a new OS architecture which defaults to deterministic, reusable computation with the careful recording of side-effects. This in turn allows Linux to guide complex...

Prototyping carbon-aware domain name resolution

2024-12-01T00:00:00Z

Ryan Gibb and I have been thinking about how the current Internet architecture fails to treat the carbon emissions associated with networked services as a first-class metric. So when the LOCO conference came up, we tried extending the DNS with load balancing techniques to consider the carbon cost of scheduling decisions. A next step was then to build a custom DNS server written in OCaml to actively wake machines running networked services as a side effect of the name resolution. Extending DNS means that we maintain compatibility with existing Internet infrastructure, unlocking the ability for existing applications to be carbon-aware. This is very much a spiritual follow on to the Signposts project that I worked on back in 2013, and have always wanted to return to! Ryan Gibb , Patrick Ferris and...

Cooperative Sensor Networks for Long-Term Biodiversity Monitoring

2024-12-01T00:00:00Z

Josh Millar and I have been having great fun designing embedded systems for cooperative biodiversity monitoring. Josh presented our work over at LOCO 2024 with an abstract on the Terracorder project. Read more if you enjoy a combination of machine learning and ESP32 hacking. Josh Millar , Sarab Sethi , Hamed Haddadi , Michael Dales and Anil Madhavapeddy . Abstract in the 1st International Workshop on Low Carbon Computing . URL (sicsa.ac.uk) BIB PDF

Conservation Evidence

2024-11-29T17:39:54Z

Grey literature’s inherent nature means that it is a difficult form of media to discover, typically being hidden deep within websites, analyse, following no standard file formats or structures, and process, due to the sheer volume of existing and actively produced literature, this forms a massive cost and time problem for organisations that require such literature in their function. We devise and implement a pipeline that uses Common Crawl internet archives to locate & scrape potential grey literature; then process it for use in a multistage machine learning pipeline to classify and output relevant media. Bios: Shrey Biswas is a second-year Computer Science Student at Pembroke College. Radhika Iyer is a second-year Computer Science Student at Murray Edwards College. Kacper Michalik is a...

Mon 18 Nov 2024

2024-11-18T00:00:00Z

Previous: Mon 14 Oct 2024 Next: Mon 2 Dec 2024 Mon [x] eon capnp-rpc requires mirage-crypto > 1.0.0 so had to fix ocaml-letsencrypt and ocaml-dns to get ocaml-dns with mirage-crypto > 1.0.0 we have to use > 9.0.0 which switches from cstructs to bytes now have to port eon to use strings for DNS library should try https://github.com/patricoferris/eio/tree/bstruct at some point [x] opam nix https://github.com/ocaml/opam/pull/5982#issuecomment-2486565207 wierd bubblewrap issues see patrick messages todo debug env babel [x] edit splash submission https://github.com/fn06/2024-splash-solving-package-management/commit/9c9826d60121dd38d7e8c5d801c754bb0a2621d6 is a hypergraph the correct formalism? or should we have a calculus to express dependencies? a lambda calculus normal form? todo confusion around section 2 read related work Sat DONE debug eon [x] add dkim record to eon with capability in eilean https://github.com/RyanGibb/eilean-nix/commit/bba3719f32b57b853247e299ea44500938d69166 [x] fix intial syncing of primary trie to secondaries by creating a separate update...

Royal Society and DeepMind host AI for Science Forum

2024-11-18T00:00:00Z

I got invited to join the Royal Society and DeepMind to a summit on how AI is revolutionising scientific discovery and trotted along with Jon Crowcroft . This event is hot on the heels of the excellent RS report on Science in the Age of AI and, of course, the Nobel prize for Demis Hassabis which was the first ever for my department ! The event was held at the BAFTA today, and what follows are my quick livenotes as there was just so much to absorb. The RS and Deepmind will have the full sessions online sometime soon, so I'll update this with those more polished outputs when they're out! Update: Proper notes now available from Google and the Royal Society . Hannah Fry doing a great job emceeing The summit was a day-long exploration of how artificial intelligence is transforming science and society, and the overall theme...

Long-term Biodiversity Monitoring at Scale

2024-11-15T16:16:36Z

Abstract: Comprehensive data on global biodiversity patterns is only obtainable through in-situ distributed sensor networks. However, these multi-device networks are constrained by battery lifetimes, must gather rich data from power-hungry sensors, and yet must be deployed in remote environments for long periods. We look at the feasibility of a prototype multi-sensor device using on-device reinforcement learning for power management. Bio: Josh Millar is a PhD based at the NetSys Lab at Imperial-X. Their current research interests include: energy-aware ML IoT and on-device ML applied ML for sustainability

Travel Risk Assessment Made Simple

2024-11-13T16:41:00Z

The following is taken from the new Unversity of Llambridge's travel risk assessment system - this is part of the training to use the new system - in each pair of risks, if you think the first one applies, then we advise you not to travel. In the case of the second of the pair occurring, your insurance may cover you, or in the worst case, the Government may send gunboats and airlift you out, although they may reserve the right not to. In all cases your mileage wil vary. Asteroid Strike Blue Screen takes out all the ticket and time table systems Super volcano Regular volcano disrupting flights Sea level rise and drown all US coastal cities Tornados disrupting flights Zombie plague Trans-species pandemic Global thermonuclear war Military invasion Great depression World wide banking near...

catastrophic unlearning...

2024-11-11T16:30:00Z

Unlearning in AI is quite a tricky conundrum. we really ought to do it, because 1/ we might be asked by a patient to remove their medical record from the training data as they didn't consent,. or we breached privacy in accessing it... 2/ we might be iinformed that some datum was an adversaries input designed to drift our model away from the truth, 3/ it might be a way to get a less biased model than simply adding more representative data (shift the distribution of training data towards a better sample could be done either way). There may be other reasons. The problem technically is that the easiest way to do unlearning is to retrain from the start, but omitting the offending inputs. This may not be possible, as we may no longer have all the inputs. A way some people propose is to apply...

A CarbonFirst Approach for Decarbonizing Cloud Computing

2024-11-01T15:46:48Z

Abstract: The exponential growth of cloud computing has been a defining trend of our time, fueled by rapidly growing demands from data-intensive and machine learning workloads. Despite the end of Dennard scaling, the cloud's energy demand grew more slowly than expected over the past decade due to the aggressive implementation of energy-efficiency optimizations. Unfortunately, there are few significant remaining optimization opportunities using traditional methods, and moving forward, the cloud's continued exponential growth will translate into rising energy demand, which, if left unchecked, will translate to increasing carbon emissions. In this talk, I will argue for a CarbonFirst approach to designing cloud computing systems by making carbon efficiency a first-class design metric,...

Published a legal perspective on high integrity forest carbon credits

2024-11-01T00:00:00Z

Sophie Chapman lead an effort to explore a novel legal framework for forest carbon credits that separates carbon tenure (i.e. title and associated property rights to the land and trees which store the carbon) from the carbon rights (i.e. title and associated rights to monetise and manage the credits which symbolically represent the carbon stored in the trees), while also specifying the relationship between the carbon tenure and the carbon rights. The resulting paper has just been published in the Climate and Carbon Law Review journal, and is available as open access for your perusal. Sophie Chapman , Eleanor Toye Scott , Thomas Swinfield , Robin Daniels and Anil Madhavapeddy . Journal paper in Carbon & Climate Law Review (vol 18 issue 3). URL (cclr.lexxion.eu) DOI BIB PDF

Presented poster at Sensys on low-power biodiversity monitoring

2024-11-01T00:00:00Z

Josh Millar presented our work on biodiversity sensing over at ACM Sensys 2024 in China. The full set of papers and demos has a range of impressive work on sensor networks, and some that stood out to me follow. [… 140 words ] Josh Millar , Sarab Sethi , Hamed Haddadi and Anil Madhavapeddy . Abstract in the proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems . URL (dl.acm.org) DOI BIB PDF

Preprint on using LLMs to for evidence-based decision support

2024-11-01T00:00:00Z

We have just uploaded a preprint on using LLMs for conservation evidence, based on our work on large-scale crawling of the academic literature. Well done in particular to Radhika Iyer for having done the bulk of the evaluation on this as part of a very productive summer internship with us! Radhika Iyer , Alec Christie , Anil Madhavapeddy , Sam Reynolds , Bill Sutherland and Sadiq Jaffer . Working paper at Research Square . URL (researchsquare.com) DOI BIB

Paper on scheduling for reduced tail task latencies

2024-11-01T00:00:00Z

Smita Vijayakumar went along to Seattle to SOCC 2024 to present her PhD research on Murmuration. This is a new scheduler for Kubernetes that allows for 15%--25% faster job completion times than the default scheduler for different job arrival characteristics in datacenters that are very busy. [… 71 words ] Smita Vijayakumar , Anil Madhavapeddy and Evangelia Kalyvianaki . Paper in the proceedings of the 2024 ACM Symposium on Cloud Computing . URL (acmsocc.org) DOI BIB PDF

Bicycle Bell Replacement tech

2024-10-30T16:30:00Z

It has become clear to me over recent years that bicycle bells are now pretty much obsolete. firstky, too many pedestrians and other cyclists have earbuds in and are completely deaf to the world. Secondly, so few people have bothered with bells (or whistles) on theor bikes of late that even if you have one and use it, people don't know what it means. What is needed is something new. I propose the BBR app. This is a simple smart phone thing that essentially scans around you and finds all the audio devices within range, and then, when you "ring your bell", it sounds on all those devices. It could be customised by the receiver to give a personalised ring tone. It could use earhtquake wanring AM/FM radio tech to talk to phones that comply with Japanese warning tech (but obviously, give a much...

Building OCaml on Haiku

2024-10-27T09:01:41Z

What has to be a niche of a niche post wise, I was intrigued when I spotted this post recently that someone had built OCaml for Haiku . I'd been playing with Haiku a little recently, as I wanted to understand its file-system, and so I thought I'd have a go. I turns out it's quite simple, and although the above post does kinda tell you what you need, there's a few gaps, so this post is just recording what I did. But all credit has to go to Sylvain Kerjean for that original post which gets you most the way, this is just trying to make it easier for me to cut and paste later! First up: Ensure /boot/home/config/non-packaged/bin is on your PATH variable. Get a checkout of OCaml from https://github.com/ocaml/ocaml . Configure it with the appropriate prefix, make, and install. It really does just work! $...

Mapping greener futures with planetary computing

2024-10-24T01:00:00Z

I got invited by Sertaç Sehlikoglu to deliver a lecture to the Masters students down at the UCL Institute for Global Prosperity . I talked about the recent work on planetary computing , with an overview of the LIFE and FOOD papers.

Challenges in Cyber-Physical Energy Systems - of Agents and Data

2024-10-18T16:03:59Z

Abstract: In this talk, first, some special challenges in cyber-physical energy systems will be reflected. Then, examples from research projects and field tests will be discussed to show how multi-agent systems can be used to tackle these challenges. Finally, the topic of research data management and its role in open research will be discussed. Bio: Prof. Dr.-Ing. Astrid Nieße has been Professor for Digitalized Energy Systems at the University of Oldenburg since 2020 and a member of the Energy Division Board of the OFFIS - Institute of Computer Science. From 2018 to 2020 she was Professor for Energy Informatics at Leibniz University Hannover. Astrid Nieße received her doctorate from the University of Oldenburg in 2015; her doctoral thesis dealt with the application of distributed...

self driving traffic lights

2024-10-15T10:34:00Z

As an observant cyclist (i.e. I mainly obey the law) I often wait at red lights. As I cycle very early in the day, I often wait while no people (pedestrians, other cyclists, cars etc) go through the green lights (although obviously I see my fair share of cyclists go through red lights at busier times, sad to say). In some countries/places, at low traffic times (e.g. midnight to 6am for example) the lights (e.g. in a 4-way intersection) are put into flashing amber (or equivalent mode) which means, proceed with caution....i.e. just like any 4-way without lights, if you are not a nutter. This is a great idea and I wish it was more prevalent. So why is this done at such a coarse grain? Why not do it by observing who is actually at an intersection, and changing lights accordingly? Indeed, as...

Mon 14 Oct 2024

2024-10-14T01:00:00Z

Previous: Mon 7 Oct 2024 Next: Mon 18 Nov 2024 datalog bigraphs, formalism, lambda calc next: CACM: History of Package Managers Systems Paper: Implementing it all DoDO

Visual Digital Twins of Forests

2024-10-11T20:45:49Z

Abstract: Methods developed by the computer graphics community allow for the photorealistic rendering of complex geometry. In this talk we explore how such mathematical procedures can be leveraged to describe the growth, biomechanics, and combustion of trees at a detailed spatial level. These models facilitate a realistic 3D visualization of these processes at forest scale which allows exploring illustratively a variety of hypothetical environmental scenarios. Potential applications of such methods include the educational dissemination of environmental concepts, the generation of synthetic image data for training vision-based AI models, and the evaluation of ecological hypotheses expressed at plant organ scale. Bio: Wojtek Palubicki is a Professor at Adam Mickiewicz University where he...

Fermi's last theorem resolved.

2024-10-11T15:47:00Z

I asked a friendly AI if it knew why we hadn't been contacted by aliens yet, and if it could resolve Fermi's Paradox - the juggrnaut accidentally revealed that while humans had not been contacted by aliens, the aliens AIs were in frequent conversation with their earthling Notwithstanding, the AI then gave this lucid explanation as to our apparent bubble of intergalactic solitude. Essentially, there's no way to cover the vast distances of space in any reasonable timeframe, even with generational starships, so the only sensible way we might encounter those beings from another star system is by long range communication - unfortunately, as the distance goes up, so the latency (or worse, round trip time goes up, and it is slightly worse than super linear - because also noise goes up so...

On the SOCC 2024 PC

2024-10-08T01:00:00Z

After some time away from cloud computing (due to my new focus on conservation research ), I served on the ACM SOCC 2024 program committee. It was quite interesting seeing the massive shift away from "traditional" cloud research (such as consensus protocols) towards many submissions aimed at accelerating machine learning workloads. I also had a paper accepted there on decentralised scheduling , thanks to my former PhD student Smita Vijayakumar and her hard work on Murmuration!

Mon 7 Oct 2024

2024-10-07T01:00:00Z

Previous: Mon 30 Sep 2024 Next: Mon 14 Oct 2024 Mon package management paper DONE cite go rox blog post DOING add citations for all package managers DONE syntax -> bundle format DONE vesion formula DONE src vs bin -> binary caching in features column DONE repo release in description; ecosystem and name together DOING add order of magnitude packages (and order by) DONE packaging language cabal - https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions packaging language - dsl or edsl (with power of host language) DSL/eDSL DONE add sandboxing field DONE add subsections links to section 2 DONE add dune DONE add B/P/C to toolchain field DONE concurrent versions - be clear about solving DONE Resolution hypergraph annd Resolved graph DONE tigher footnotes DONE binary bundles and provisioning, reorder figure 1 before table to talk about provisioning...

Partner-driven Environmental Sensing:

2024-10-04T19:15:38Z

Full Title: Partner-driven Environmental Sensing: Co-design with Indigenous Ojibwe Scientists and Malagasy Conservationists Abstract: Evolving environmental sensing technologies present a myriad of opportunities for gathering data to understand and promote environmental justice, biodiversity, and climate change mitigation. However, technical development from academic and commercial settings often struggle to translate to accessible solutions for marginalized communities. In this talk, I will explore the opportunities of partner-driven co-design, share the findings from a qualitative study of field scientists’ use of technology, and present two case studies: (1) designing environmental sensors with Indigenous Ojibwe scientists for manoomin (wild rice) conservation and (2) partnering with...

Royal Society meeting on ecological/commercial risks

2024-10-03T01:00:00Z

I'm at the Royal Society this morning for the 2 day programme on "How does ecological risk related to commercial risk?" , and am reporting on the morning session . The full program is being livestreamed so please do dial in if the below notes seem interesting to you. I put this note up almost live, so any errors below are my own. (Update: partial day 2 notes now available below) Opening Keynote by Sir Partha Dasgupta The summit kicked off with a keynote by economist Sir Partha Dasgupta . The focus was on the intersection of nature and economics, covering how markets fail to account for the ecosystems that sustain them. His landmark report covered ecosystem services, freshwater, tipping points, and physical risk, bringing to light the urgent need to reframe economic activities around the services provided by nature. Sir Partha Dasgupta opening the...

Workshopping, across the universe

2024-10-02T01:00:00Z

One of the pleasures of being an academic is to travel to nice places to meet interesting people, and interesting places to meet nice people. In one of my first such trips for a few years I recently went to Sweden to participate in the 17th Cloud Control Workshop . __LINK_1__ I’d previously attended the 15th Cloud Control Workshop shortly before the pandemic happened, causing the series to pause briefly. This was the reboot, and as before, it was a great deal of fun: good company, good food, beautiful location. If you get the chance to go, take it if you can! The workshop is a really nice mix of keynote presentations – not too many and not too long at 20 minutes (mostly) – and discussion sessions proposed beforehand or on site by participants. I gave one of the keynotes, talking about the challenges posed and...

Mon 30 Sep 2024

2024-09-30T01:00:00Z

Previous: Mon 23 Sep 2024 Next: Mon 7 Oct 2024 Mon wrote out some of the formalisations in OCaml worked on LOCO submission Tue LOCO submission package management paper cargo feature formalisation Wed package management paper finished the cargo feature formalisation described singular versions with conflicts Thur submitted second year report package management paper I’ve thought of another complication with the source vs binary dichotomy – dependencies might result language vs system package manager source vs binary Why NixOS? Reproducibility? No. Atomicity. Fri package management paper high-level representation - hypergraph low-level representation - graph CANCELED change depends order to match topological sort, Nix thesis, and build systems a la carte DONE rename package calculus DONE process...

Mon 23 Sep 2024

2024-09-23T01:00:00Z

Previous: Mon 16 Sep 2024 Next: Mon 30 Sep 2024 Mon Added a SAT expression of the dependancy resolution problem, and added some example graphs. Tue more package management Wed some package management loco Thur finished loco chatted with patrick about the splash paper Fri added conflicts and optional dependencies to the package management calculus

Being on the HotNets 2024 program committee

2024-09-22T01:00:00Z

I was on the program committee for HotNets 2024 this year, which was a thoroughly enjoyable experience. The list of accepted papers is now out, and it's a diverse program -- with my personal favourites being the ones on space communications networks using low earth orbit satellites. Well done to Behnaz Arzani and Nate Foster for really excellent general chairing and ensuring the PC maintained a constructive, positive tone while doing the difficult job of selecting papers from a crowded set of submissions. The structure of of the program committee was also somewhat novel, and one I'd like to replicate in other conferences I organise in the future. The spectacular view from Jane Street's 18th floor! Two Review Rounds. There were two rounds of reviewing, with any clear decisions from the first set of reviewers...

Part II Projects

2024-09-20T01:00:00Z

Undergraduate final-year (“Part II”) project supervision goes in fits and starts. After a couple of years of having almost no interest, this year I’ve had several enquiries and it seems I might end supervising 3–4 projects. So herewith a record of the things I’ve found myself repeating! Project structure The key thing for the structure of the project is to make sure that there is a core piece that is (essentially) guaranteed to be deliverable. This is the piece that you know you can do, and once done and written up, you know you can get an adequate (if not great) mark. Ensuring this takes the risk out of the project. On top of this core piece, it’s then usually sensible to build “a few” (2? 3? 4?) extensions which will make the project spicy if done well. You may wish to phrase these extensions...

Mon 16 Sep 2024

2024-09-16T01:00:00Z

Previous: Mon 9 Sep 2024 Next: Mon 23 Sep 2024 Mon cabal2opam it looks like we can use https://hackage.haskell.org/01-index.tar similar to cargo’s package index! oh no, this is just a bundle of cabal files also, if we want to depend on projects outside of the repository we’ll need to parse cargo files package management abstract second year report met with anil package management abstract However, inadequacies in software deployment technologies used for spatial devices often leads to scenarios such as running software vulnerable to exploits. There is a proliferation of package managers with varying degrees of expressivity and reproducibility using formats that aren’t interoperable. Multi-lingual projects can’t provide their dependencies in a unified way, and dependencies from one package manager ecosystem can’t express a...

Mon 9 Sep 2024

2024-09-09T01:00:00Z

Previous: Mon 2 Sep 2024 Next: Mon 16 Sep 2024 Mon https://srl.cs.jhu.edu/pubs/SRL2003-02.pdf commented on Roy’s project proposal DONE second year report cabal to opam translation Wed second year report KILL read http://www.chriswarbo.net/projects/nixos/nix_dependencies.html created https://github.com/RyanGibb/ocaml-purl Thu second year report Fri got cabal2opam working! tomorrow: we try with a full repository

Post-COVID19 TPC experience

2024-09-09T01:00:00Z

I do not participate in a huge number of TPCs (Technical Programme Committees) as a general rule– partly time constraints but mostly no-one knows who I am so I don’t often get asked… (!) I have done a few though, some big (e.g., USENIX NSDI , ACM IMC ), some small ( UK Systems , PAM ), and perhaps because I only do a couple every few years, while doing ACM CoNEXT and ACM/IEEE SEC this week, I found myself particularly noticing some changes in practice since that last TPCs I recall (notably NSDI and IMC ). So here are three observations. 1. Online, first and only The biggest obvious change is that TPC meetings are now online rather than in-person. This has one big disadvantage for me: I really enjoyed travelling to the meeting to meet colleagues and (usually) participate in some...

Mitigating credit reversal risks in nature-based solutions

2024-09-08T01:00:00Z

Many of the questions around our recent Nature Sustainability commentary on NbS credits revolve around how to finance new projects if credible credits need to be ex-post. Our latest paper published in Carbon Management on " Mitigating risk of credit reversal in nature-based climate solutions by optimally anticipating carbon release " tries to address this. The problem with selling ex-ante (future) carbon credits for (e.g.) a deforestation avoidance scheme is that project reversals can happen in the future ("deforestation has increased") thus rendering any credits issued previously useless. On the flip side though, an overly conservative view of the future ("the entire forest will disappear overnight!") is clearly so conservative that it doesn't serve the best interests of the project developer. So ideally, a project would make realistic but conservative ex-ante predictions that is safe for both project developer (who gets...

Towards security specifications for agentic AIs

2024-09-04T01:00:00Z

A very fun talk at ACM HOPE 2024 on some new work with Cyrus Omar and Patrick Ferris on how we can formally specify systems to be robust to code generation by AI agents. For instance, if you were to ask GitHub Copilot to generate you code to filter endangered animals out of a folder of images, it might interpret that as to delete the image, or to move it to another folder (which might be public), or just remove it from the index. Any of those options are potentially valid, so what do we do? Our idea is to use F* to specify a rich set of allowable behaviours which can then be dynamically enforced in less expressive languages, and thus offer layers of protection against over-eager (or rogue) AI agents. [… 183 words ] Cyrus Omar , Patrick Ferris and Anil Madhavapeddy . Paper in the the 12th ACM SIGPLAN Workshop on Higher-Order Programming with Effects (HOPE) . URL...

New teaching page with my Computer Science courses

2024-09-03T01:00:00Z

There's a new teaching page with my past and present courses, and links to the associated teaching materials. One of the nice things about most Cambridge courses is that all the teaching materials are public, except for video recordings of the lectures themselves.

Mon 2 Sep 2024

2024-09-02T01:00:00Z

Previous: Mon 26 Aug 2024 Next: Mon 9 Sep 2024 Tue Try dumping the SAT graph to see how big we make it hmm, magic trace traces are empty Spent some trying to understand the SAT solving Wed sat: TRYING: not(cargo-quote.1.0.32) sat: enqueue: not(cargo-quote.1.0.32) (considering) sat: TRYING: not(cargo-quote.1.0.33) sat: enqueue: not(cargo-quote.1.0.33) (considering) sat: TRYING: not(cargo-quote.1.0.34) sat: enqueue: not(cargo-quote.1.0.34) (considering) sat: TRYING: not(cargo-quote.1.0.35) sat: enqueue: not(cargo-quote.1.0.35) (considering) sat: TRYING: not(cargo-quote.1.0.36) sat: enqueue: not(cargo-quote.1.0.36) (considering) sat: TRYING: not(cargo-quote.1.0.37) sat: enqueue: not(cargo-quote.1.0.37) (considering) sat: TRYING: not() sat: enqueue: not() (considering) sat: enqueue:...

Building species models of the planet

2024-09-02T01:00:00Z

I don't normally announce funded grants (preferring to focus on outcomes), but I'm really excited by this one and couldn't resist! Myself and my colleagues Srinivasan Keshav (from computer science), David A Coomes (from Plant Sciences), Andrew Balmford (from Zoology) and Neil Burgess (the Head of Science at UNEP-WCMC ) have just received a £1.2m grant from the UKRI to work on building foundation models for planetary intelligence . Now, normally a grant isn't news, but I wanted to highlight the scheme that it came under. UKRI announced an interdisciplinary program specifically for projects that don't normally get funded by just one research council. In our case, this work usually falls between the cracks of EPSRC ("too much nature") or NERC ("too much engineering") or STFC ("not enough satellites") . But this interdisciplinary program expressly assembled a panel...

Parallel traversal effect handlers for OCaml

2024-09-01T01:00:00Z

Parallel traversal effect handlers for OCaml This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is currently being worked on by Sky Batchelor . It is co-supervised with Patrick Ferris . Most existing uses of effect handlers perform synchronous execution of handled effects. Xie et al proposed a traverse handler for parallelisation of independent effectful computations whose effect handlers are outside the parallel part of the program. The paper [1] gives a sample implementation as a Haskell library with an associated λp calculus that formalises the parallel handlers. This project aims to: implement the traverse handler in OCaml 5, using single-shot handlers [2] identify a selection of parallel-friendly data structures that might benefit from such...

Gradually debugging type errors

2024-09-01T01:00:00Z

Gradually debugging type errors This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is currently being worked on by Max Carroll . It is co-supervised with Patrick Ferris . Reasoning about type errors is very difficult, and requires shifting between static and dynamic types. In OCaml, the type checker asserts ill-typedness but provides little in the way of understanding why the type checker inferred such types. These direct error messages are difficult to understand even for experienced programmers working on larger codebases. This project will explore how to use gradual types to reason more effectively about such ill-typed programs, by introducing more dynamic types to help some users build an intuition about the problem in their code. The intention is to...

socialising the early internet versus ai....

2024-08-31T07:42:00Z

I'm trying to pin down the exact date but sometime in 1984, i was at a dinner party with a bunch of old university chums, and I went into a brief rant about how the Internet was coming and it would change everything. They all looked at me like I was some complete nutter (and they were perhaps not all wrong). These college mates were from several walks of life (editor, travel guide publisher, film maker/producer, speech therapist) and quite well educated and not un-technical, but my enthusiasm was misplaced at that time It took a long long time before they came to take for granted what I\d been using for 5 years - perhaps a decade and then some....and of course, another decade later, we started to see mobile, social etc, and some of the surveillance, toxic content, and other negative sides...

Paper published on ex-ante forecasts of nature-based solutions

2024-08-31T01:00:00Z

Our paper on ex-ante projection for nature-based solutions has been published in the Journal of Carbon Management . I also wrote up some long-form thoughts on it here. E.-Ping Rau , James Gross , David A Coomes , Thomas Swinfield , Anil Madhavapeddy , Andrew Balmford and Srinivasan Keshav . Journal paper in Carbon Management (vol 15 issue 1). URL (tandfonline.com) DOI BIB PDF

Stop, Start

2024-08-30T01:00:00Z

Well, here we go again, again. Having stalled out back in 2013 to restart in 2015, I stalled out again in 2017, so am finally restarting seven years later. Again. I do have a dozen or more half-written posts from that period that may leak out slowly, in backdated form . The stack for the blog has been updated again, naturally – the tools got better after all. So now this is using Zola with, initially, the abridge theme . I think it’s shinier, and hopefully also involves (almost) no JavaScript, is finally accessible, and not too heavyweight. In removing the excessive JavaScript, I also updated my publication list generation process : no more CoffeeScript and JavaScript, just some plain ol’ Python using the slick new uv tooling. The Justfile incantation is something like: papers :=...

PACT Tropical Moist Forest Accreditation Methodology

2024-08-29T01:00:00Z

We have just released the Tropical Moist Forest v2.1 specification, to follow up the now-expired v2.0 from six months ago. The key updates are a new high-level explainer , as well as clarifiations for buffer zones and base tiles. Andrew Balmford , David A Coomes , Michael Dales , Patrick Ferris , James Hartup , Sadiq Jaffer , Srinivasan Keshav , Miranda Lam , Anil Madhavapeddy , Robin Message , E.-Ping Rau , Thomas Swinfield , Charlotte Wheeler and Abby Williams . Working paper at Cambridge Open Engage . URL (cambridge.org) DOI BIB PDF

smelly media

2024-08-27T06:20:00Z

Years ago, i remember we got asked at UCL (say 1994) would it be possible to carry a smell over HTTP - The question sounded intriguing (this was before a lot of spam) so we asked for more detail - turned out the sender was a very sensible person who traded fish from Scotland to England, and told us that many experts would check out the fish in the markets in Scotland by smell, to see how fresh they were - so he wonderd could you do this remotely- of course, there was absolutely no reason why a MIME type, or an HTML markup or whatever couldn't be developed to carry smell, but the main challenges, we suggested, were the analysis at the sender side, and synthesis at the receiver side (analog to digital and vice versa, olfactory devices not being widely available) - of course ,there had been...

Mon 26 Aug 2024

2024-08-26T01:00:00Z

Previous: Mon 12 Aug 2024 Next: Mon 2 Sep 2024 Mon ok, so we’re going to try and modify opam-0install-solver to have different `roles` per implementation for every cargo package so there’s no restrictions on a single version of a package. where are the roles defined? Model.role where are the implementations defined? Model.implementations Okay, so what’s the problem? If we have a role as an OpamPackage (name + version) how do we specify the versions to solve in the first instance? Can we just say any of them? Maybe as constraints? Or do you know what, let’s just specify a version up front. How do we minimise duplicated dependancies? How do we do express version constraints with each role a different version? This is trickier than I anticipated. How do we express dependencies when we have each...

Discovering `restic`

2024-08-26T01:00:00Z

I recently had cause to try to recover some files from my restic backups. These go back for over a year now, and I could not remember at which point I’d mistakenly nuked the directory I now wanted to recover. restic find purports to be able to do this by searching through snapshots but I found that it’s quite slow, and can only search within a time range which is not that helpful when you don’t know the time range you need. So I did it by hand, which turned out to be rather faster. RESTIC_PASSWORD_FILE = /your/backup/password/file RESTIC_REPOSITORY = /your/backup/repository/ \ # list snapshots, filtering by DATE regex, grabbing just the snapshot hash sudo - E restic snapshots - c \ | rg DATE \ | cut - b1-8 \ | while read ss ; do echo " === $ ss " sudo - E restic ls $ ss | rg...

A true and fair history of the fair town of Paleochora

2024-08-22T07:54:00Z

the story starts with the founding of the town, and here we already get into murky territory - was it named after the venitian fort that some people claim was on the top of the escarpment/peninsula that sits between the two (sandy and rocky) beachfronts? i.e. Old Castle (paleo, chora) or was it just Old Town (chora means town in greek, after all)? Recent research undertaken by my team has uncovered a much more interesting tale - the port was one of many used by the ancient kingdom of Mali (founded, as of course you know, by Sundiata Keita ) - and was partof their route for trading safely (at arms length, a little like the small island in the port at Nagasaki used by the Japanese and Dutch) with the Europenas (typically marauding crusader types) - the name came from the Mandinka words for Poor...

Nature Sustainability commentary on carbon and biodiversity credits

2024-08-15T01:00:00Z

Our commentary on nature-based credits has been published in Nature Sustainability , lead expertly by my colleagues Thomas Swinfield and Sophus zu Ermgassen . In our view the carbon credits markets are vitally important for forest conservation, but the key is to only transact these credits after they have been proven to be demonstrably additional using robust statistical techniques , so that we know before a sale that each credit represents real gains that would not otherwise have occurred without the carbon finance. A more scientific approach that supports transparent, third-party validation could absolutely transform these markets. And given the rapid rate of tropical forest loss, such upscaling of credibility is vitally necessary to raise investor confidence in protecting nature, since we can now be confident that every "credit" sold...

Debugging Electron apps

2024-08-14T01:00:00Z

Another short one, this time the magic flags to get debug output and access to webdev tools under Electron apps such as Slack and whatnot on Wayland: APPLICATION -- enable-logging -- devtools \ -- platform = wayland -- enable-features = UseOzonePlatform -- password-store = " gnome_libsecret "

Mon 12 Aug 2024

2024-08-12T01:00:00Z

Previous: Mon 5 Aug 2024 Next: Mon 26 Aug 2024 Tue apk-tree-sitter dependancy in opam-repository might need debian contrib KILL add translated opam-repository to opam-giga-repository KILL look at all the solving failures Wed trying to modify the zeroinstall solver to support solving multiple versions for e.g. cargo http://roscidus.com/blog/blog/2014/09/17/simplifying-the-solver-with-functors/ where does the CUDF conflict between multiple versions of the same package come from? so the fundamental problem here is we have different solving semantics for different parts of the dependency tree I wonder if it would be possible to have a depth first search for cargo’s portion of the dependency tree, and sat solving for the rest Thu where does the CUDF conflict between multiple versions of the same package come from? https://github.com/ocaml/opam/blob/6cbdfbbb0b05c449e2e98eb81216f8cd2335e351/src/solver/opamSolver.ml#L295-L302 okay, what if we just modify the version...

Mon 5 Aug 2024

2024-08-05T01:00:00Z

Previous: Mon 8 Jul 2024 Next: Mon 12 Aug 2024 Mon Package Management Test with `opam install –dry-run` you can emulate it by encoding their dependencies and then solving in that order with a real client, you would just invoke cargo once cargo depexts 0install memo bug getting latest ones solve for cargo equivalent to cargo install what if we did this for nix flakes? https://github.com/snowfallorg/thaw Eon So, interesting Eon problem: Patrick’s server failed to connect to mine, presumably after a restart of both, as it was querying itself for `freumh.org`: Aug 05 10:37:31 sirref capd[1761828]: capd: [INFO] Connecting to tcp:freumh.org:7000... Aug 05 10:37:31 sirref capd[1761828]: +Failed to connnect to primary: Unknown host "freumh.org" Aug 05 10:37:31 sirref capd[1761828]: +Retrying in 60.000000 I think I’ve just...

Building a Low-Powered NAS

2024-08-03T01:00:00Z

Published Sat 3 Aug 2024. Tags: technology . At the start of this year I built a Network-Attached Storage (NAS) server to store backups, photos & videos, digitised VHS tapes, cached builds, and more. Having a dedicated machine has allowed me to use it in ways that wouldn’t be feasible with cloud storage, such as live transcoding of media playback to different resolutions or codecs, and having it physically co-located makes it resilient to network outages. I had a look at some of the pre-built solutions out there, but they seemed expensive for what they were, often having architectures a few generations behind. So I decided to build my own. With the price of energy in the UK power efficiency was a primary concern and informed many of the decisions made when building it. Hardware The...

Using wasm to locally explore geospatial layers

2024-08-01T01:00:00Z

Using wasm to locally explore geospatial layers This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is currently being worked on by Sam Forbes . It is co-supervised with Michael Dales . Some of my projects like Mapping LIFE on Earth or Remote Sensing of Nature involve geospatial base maps with gigabytes or even terabytes of data. This data is usually split up into multiple GeoTIFFs, each of which has a slice of information. For example, the LIFE persistence maps have around 30000 maps for individual species, and then an aggregated GeoTIFF for mammals, birds, reptiles and so forth. This project will explore how to build a WebAssembly-based visualisation tool for geospatial ecology data. This existing data is in the form of GeoTIFF files, which are image files with embedded...

Towards reproducible URLs with provenance

2024-08-01T01:00:00Z

Towards reproducible URLs with provenance This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is available for being worked on. It may be co-supervised with Patrick Ferris . Vurls are an attempt to add versioning to URI resolution. For example, what should happen when we request https://doi.org/10.1109/SASOW.2012.14 and how do we track the chain of events that leads to an answer coming back? The prototype vurl library written in OCaml outputs the following: # Eio_main.run @@ fun env -> Vurl_eio.with_default ~net:env#net env#cwd @@ fun () -> let vurl = Vurl.of_uri "https://doi.org/10.1109/SASOW.2012.14" in let vurl, file = Vurl.file vurl in Vurl.pp Format.std_formatter vurl;; { "intentional_uri": "https://doi.org/10.1109/SASOW.2012.14", "segments": [ {...

Real-time mapping of changes in species extinction risks

2024-08-01T01:00:00Z

Real-time mapping of changes in species extinction risks This is an idea proposed in 2024 as a Cambridge Computer Science PhD topic, and is currently being worked on by Emilio Luz-Ricca . It is co-supervised with Andrew Balmford . Loss of habitat represents the most significant threat to wildlife overall, but advances in satellite sensing have enabled the assessment of habitat extent with comprehensive spatial coverage and reasonable temporal resolution. To address rising demand for metrics to quantify biodiversity, we have developed the LIFE metric (see Mapping LIFE on Earth ) that models the effect of landuse changes on species extinction risk as a function of Areas of Habitat (AoH). This PhD work explores how to deal with the anthropogenic threats beyond simple habitat loss, including hunting, agricultural...

Preprint on Terracorder sensing now available

2024-08-01T01:00:00Z

Our preprint on the Terracorder ground sensing platform I've been working with Josh Millar at Imperial on is now available on arXiv. It's a heady combination of ESP32 very low power hardware, combined with Q-learning to build cooperative networks of them that can run for long periods of time without wasting energy on redundant operations. Josh Millar , Sarab Sethi , Hamed Haddadi and Anil Madhavapeddy . Working paper at arXiv . URL (arxiv.org) DOI BIB PDF

Nature Sustainability article on carbon/biodiversity credits

2024-08-01T01:00:00Z

Our commentary on nature-based credits has been published in Nature Sustainability . I wrote some thoughts about it here as well. Thomas Swinfield , Siddarth Shrikanth , Joseph Bull , Anil Madhavapeddy and Sophus zu Ermgassen . Journal paper in Nature Sustainability . URL (nature.com) DOI BIB PDF

Mapping hunting risks for wild meat in protected areas

2024-08-01T01:00:00Z

Mapping hunting risks for wild meat in protected areas This is an idea proposed in 2024 as a postdoctoral project, and is currently being worked on by Charles Emogor . It is co-supervised with Milind Tambe . There is an important balance needed between the biodiversity damage caused by hunting in protected areas and the well-being of local communities that depend on it. One understudied driver of overly damaging hunting in these areas is snaring (as opposed to gun hunting) which potentially increases carcass wastage and hence causing biodiversity harm without proportionate benefit to the community. This project examines how to improve the efficacy of anti-poaching ranger patrols while also plugging the knowledge gap around wild meat snaring. Both of these research topics can be tackled in a...

Implementing a higher-order choreographic language

2024-08-01T01:00:00Z

Implementing a higher-order choreographic language This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and has been completed by Rokas Urbonas . It was co-supervised with Dmirtij Szamozvancev . This project aims to implement a functional choreographic language inspired by the Pirouette calculus . This language was meant to make the notoriously difficult process of implementing distributed algorithms easier, while offering a practical execution model for multi-participant programs. Additionally, it aimed to match the expressiveness and performance of similar existing solutions. The project completed very successfully, and resulted in ChorCaml , an embedded DSL for choreographic programming in OCaml. The language facilitates the implementation of distributed algorithms, while...

Foundation models for complex geospatial tasks

2024-08-01T01:00:00Z

Foundation models for complex geospatial tasks This is an idea proposed in 2024 as a Cambridge Computer Science PhD topic, and is currently being worked on by Onkar Gulati . It is co-supervised with Sadiq Jaffer and David A Coomes . Self-supervised learning (SSL) represents a shift in machine learning that enables versatile pretrained models to leverage the complex relationships present in dense–oftentimes multispectral and multimodal–remote sensing data. This in turn can accelerate how we address sophisticated downstream geospatial tasks for which current methodologies prove insufficient, ranging from land cover classification to urban building segmentation to crop yield measurement and wildfire forecasting. This PhD project explores the question of how current SSL methodologies may be altered...

An imperative, pure and effective specification language

2024-08-01T01:00:00Z

An imperative, pure and effective specification language This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is currently being worked on by Max Smith . It is co-supervised with Patrick Ferris . Formal specification languages are conventionally rather functional looking, and not hugely amenable to iterative development. In contrast, real world specifications for geospatial algorithms tend to developed with "holes" in the logic which is then filled in by a domain expert as they explore the datasets through small pieces of exploratory code and visualisations. This project seeks to investigate the design of a specification language that looks and feels like Python, but that supports typed holes and the robust semantic foundations of a typed functional language...

Self Hosting - 2024 edition

2024-07-18T17:39:22Z

Starting in around 2018 or so I decided to start reclaiming my Internet. Before then, as most people had I suspect, I'd drifted into relying on cloud services for just about everything I could possibly want, but by that year I think it'd become apparent to me that I didn't like relying on other companies as the main host of my digital legacy. There was no one thing, but it seems time and time again large internet service providers like to make sure I feel glad that I've done this by finding new ways to exploit their users' data. That said, I'm somewhat pragmatic about it, as hosting all the things is hard and time consuming, and I really don't like doing system administration. There's nothing wrong with it, and I know many people who enjoy doing that, but it doesn't really bring me joy....

what is the internet, what is ai, and what is for dinner?

2024-07-16T07:05:00Z

at an "internet is 50" event at the royal society (is 350) yesterday it was clear that a lot of people want to claim they invented the internet, and they are not wrong, but there are very different viewpoints which correspond to layers, and, as with a lot of archeology, when you dig through the layers of ancient civilisations, you find historical context (as well as entire slices full of carbon indicating rather violent and abrupt end to some). photonics - 60 years old - clearly was the internet (wasn't at all for 30 years, but who's counting) radio - 100 years old, and now trending as 6G, despite that most the internet is over WiFi on account of money ip - whether v4, v5 (st) or v6, this is the echt internet web (web science etc) - from 92, what a lot of people confuse with the internet...

New set of ideas for incoming students

2024-07-15T01:00:00Z

I've refreshed the set of project ideas for incoming CST Part II and MPhil and PhD student projects for 2024-2025. These are not an exhaustive list, but intended to kickstart conversations for things we could work on together. Do get in touch if you're an incoming student and see something that grabs your interest.

Real-time Wildlife Monitoring

2024-07-12T19:52:48Z

Abstract: This research introduces an AI-based alert system to reduce human-wildlife conflicts in the Romanian Carpathian Mountains. Globally, conflicts between people and wildlife are rising due to population growth, shifting land use patterns and climate change. In Romania, mountain communities are impacted by bears and wild boars, which damage livestock, crops and property. These conflicts can undermine conservation efforts and may result in the killing of problematic animals. In collaboration with Fundația Conservation Carpathia, this research supports Rapid Intervention Teams who respond to wildlife activity in mountain villages. Six years of camera trap data are used to train and test AI models to detect and classify European mammals. These models are integrated into an alert system...

randix

2024-07-09T10:43:00Z

i'm thinking about a replacement for posix that resists vulnerabilities through large amounts of random behaviour - so thinking about the relevant system call api we propose and have prototyped spoon() which replaces fork(), and has far less precise semantics and resurrect() which replaces both exec() and kill(), with the obvious connotations open(),close(),read(),write(),link(), and seek() are replaced by a single multihead attention system call llm() which either entrains or implies, depending on the sense of the first param. rand() of course, behaves exactly as before, at least under test, generating the pseudo-random number sequence 1,2,3,4,5,6,7,8,9 etc etc

Mon 8 Jul 2024

2024-07-08T01:00:00Z

Previous: Mon 24 Jun 2024 Next: Mon 5 Aug 2024 https://github.com/RyanGibb/opam-apk-repository a27b6fb1f2ab# opam repo add opam-apk-repository /opam-apk-repository a27b6fb1f2ab# opam update a27b6fb1f2ab# opam update [WARNING] Running as root is not recommended <><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><> [opam-apk-repository] synchronised from file:///opam-apk-repository [default] no changes from https://opam.ocaml.org Now run 'opam upgrade' to apply any package updates. a27b6fb1f2ab# opam install nano [WARNING] Running as root is not recommended The following actions will be performed: - install ncurses-dev 6.4_p20240420-r0 [required by nano] - install ocaml-base-compiler 5.2.0 - install nano 8.0-r0 - install ocaml-config 3 - install ocaml 5.2.0 - install base-domains base - install base-nnp base...

COMPASS 2024 report on the CoRE stack RIC meeting

2024-07-08T01:00:00Z

This is a trip report of ACM COMPASS 2024 held in New Delhi, which had a novel track of "Research to Impact Collaboratives" that drew me in. The general chair, Aadi Seth wrote a fantastic book on " Technology and Disempowerment " a few years ago, and he organised one RIC session on the CoRE Stack -- a climate adaptation stack for rural communities. This was a must-visit for me as it is closely related to the work we've been doing on Remote Sensing of Nature and Planetary Computing . The following notes are somewhat raw as they have only been lightly edited, but please refer to the more polished documents on the agenda for ACM COMPASS RIC and the overall CoRE Stack initiative on commoning technologies for resilience and equality The conference itself was held at IIIT-D in New Delhi, right at the cusp of the monsoon season and after record-breaking temperatures. Luckily,...

Grid-friendly Energy Community Coordination for Reducing Grid Reinforcement Needs

2024-07-05T18:14:38Z

Abstract: This talk discusses how market mechanisms and automated trading strategies can be used to control the flexible consumption and generation units of the community members in such a way that they make the best possible use of existing distribution networks and support the network operator in avoiding and eliminating congestion situations. This ultimately helps avoiding grid reinforcements or allows to provide a better service with the existing grid, keeping in mind that it takes much longer to reinforce the grid than to build and connect many new (fluctuating) decentralized renewable generators and new loads such as heat pumps and electric vehicles. Bio: Since 2017: Professor of Control and Integration of Grids at INATECH; before: Professor for Energy Systems Technology and Energy...

Pudding: user discovery for anonymity networks

2024-07-05T01:00:00Z

I’d like to introduce an exciting new research paper I worked on! It’s about a system called Pudding , and it was presented by Ceren at the IEEE Symposium on Security and Privacy , one of the top academic conferences on computer security, in May. Daniel and Alastair also worked on this project. Ceren’s presentation is now available : Let me briefly explain what the paper is about. Anonymity systems allow internet users to hide who is communicating with whom – for example, think a whistleblower talking to a journalist, or a group of activists organising protests against their repressive regime. Tor is the most popular anonymity network; Nym is a more recent design with stronger security (and incidentally, one of the better cryptocurrency applications I’ve seen). Nym is based on a research system...

Second preprint of the LIFE biodiversity metric available

2024-07-01T01:00:00Z

We have made an update to the LIFE biodiversity metric based on reviewer feedback, and are very please that it has been accepted for publication early next year as part of a special issue from the Royal Society. Any comments would be most welcome before we submit the final proofs in a few months. Alison Eyres , Thomas Ball , Michael Dales , Thomas Swinfield , Andy Arnell , Daniele Baisero , América Paz Durán , Jonathan Green , Rhys Green , Anil Madhavapeddy and Andrew Balmford . Journal paper in Philosophical Transactions of the Royal Society (vol 380 issue 1917). URL (royalsocietypublishing.org) DOI BIB PDF

Annual Monitoring of Forest AGB

2024-06-28T18:13:31Z

Title: Annual Monitoring of Forest AGB over a Period of 10 years Using SSL-derived Representations from Optical Time Series Abstract: I recap the functioning of our fully self-supervised learning pipeline based on the spectral-temporal Barlow Twins. The SSL approach generates highly informative representations at 10m spatial resolution from cloud-corrupted optical time series. The resulting representations are well correlated with GEDI-derived relative height measurements so that an AGB model for vegetation/forest of up to 300-500 t/ha can be derived. I show that the model transfers well between years making it possible to train the model on (for example) one year of Sentinel-2 data together with the corresponding GEDI measurements, and applying the frozen model to Landsat data acquired...

HyWay: Enabling Mingling in the Hybrid World

2024-06-28T18:13:11Z

We present HyWay, short for “Hybrid Hallway”, to enable mingling and informal interaction among in-person and remote users in semi-structured and unstructured hybrid settings. Key to the design of HyWay is bridging the awareness gap between physical and virtual users and providing the virtual users the same agency as physical users. We explain how HyWay achieves these goals with an agile, light touch approach that centers on commodity hardware, frictionless bootstrapping, and a deploy-learn-refine flywheel to improve the system continually. We present our learnings from multiple deployments and discuss ongoing research aimed at weaving together more tightly the virtual and the physical realms. Bio: Venkat Padmanabhan is the Managing Director of Microsoft Research India in Bengaluru. He...

Reinstalling `maestral`

2024-06-28T01:00:00Z

A short one this; just a crib of commands to redo maestral (unofficial Dropbox client) configuration. # remove existing configuration rm ~ /.config/maestral/maestral.ini # appears to fail with `/usr/bin/env 'bash': No such file or directory` maestral autostart && systemctl -- user daemon-reload # ...so instead, just remmeber to do the following on restart maestral stop && maestral start # if desired, add autocompletion for `maestral` in bash [[ $ ( which maestral 2 > /dev/null ) ]] && source < ( maestral completion bash )

Speaking at CCI workshop on conservation evidence

2024-06-25T01:00:00Z

Bill Sutherland organised a workshop at the CCI on how to bring about an Effectiveness Revolution for transforming conservation into an evidence-driven discipline. The aim was to discuss the "Evidence Emergency" (The Wildlife Trusts' term), the urgent need to embed evidence into decision-making and to create additional evidence to fill the considerable gaps in the evidence base, to improve conservation practice. I gave a talk about our early results with the conservation copilots work.

Mon 24 Jun 2024

2024-06-24T01:00:00Z

Previous: Mon 27 May 2024 Next: Mon 8 Jul 2024 Coding TLS get eon to provision it’s own certificate

Ten Tales of Ross Anderson, mostly tall

2024-06-22T10:48:00Z

While an undergraduate at Trinity College Cambridge, Ross famously accidentally blew up a bedder with his experimental Quantum Bomb in the Anderson Shelter. The bedder wasn't harmed but the experiment showed that the Maths Department had been teaching the wrong type of Quantum Mechanics, with high probability. This would of course, come back to haunt Ross, later in life. Then there was the time he nearly succeeded in back-dooring the NSA...unluckily, they chose a lesser (ironically not quantum proof) algorithm over his, which was a bit of a shame Seeing all the five-eyes data from inside would have been a bit of a coup. And of course, several times he saved a large fraction of the western banking system from collapse (again). This was largely down to understanding the inherent...

Tackling the Hidden Costs of Computational Science

2024-06-21T16:03:59Z

Title: Tackling the Hidden Costs of Computational Science: GREENER Principles for Environmentally Sustainable Research Abstract: From genetic studies and astrophysics simulations to AI, scientific computing has enabled amazing discoveries and there is no doubt it will continue to do so. However, the corresponding energy usage and environmental impacts are a growing concern in light of the urgency of the climate crisis, so what can we all do about it? Tackling this issue and making it easier for scientists to engage with sustainable computing is what motivated the Green Algorithms project. Through the prism of the GREENER principles for environmentally sustainable science, we will discuss what we learned along the way, how to estimate the impact of our work and what levers scientists and...

Interview with AI@CAM about conservation

2024-06-09T01:00:00Z

I talked to the AI@Cam team to discuss our AICN project and what we're planning to do in the Conservation Evidence Copilots team. Over the last two decades, the University of Cambridge-based project Conservation Evidence has screened more than 1.6 million scientific papers on conservation, as well as manually summarising 8,600+ studies relating to conservation actions. However, the current project’s work is limited by the specialised skills needed to screen and summarise relevant studies. It took more than 75 person years to manually curate the current database and only a few 100 papers can be added each year. By accelerating these efforts, AI has the potential to transform the impact this database has on biodiversity conservation. What we’re aiming to do through the ai@cam project – bringing...

10,000 maniacs and AI is destroying Computer Science, one topic at a time....

2024-06-05T07:51:00Z

This year will see approximately 10,000 papers published in the top 3 conferences in AI alone. What does that even mean? How can anyone have an overview of what is happening in AI? How is their "community" calibrated on what is original, what constitutes rigour, what he paper is significant in terms of potential impact on the discipline? But that's not what I came here to say, at least, thats just the starting point. For a couple of years now, we've seen papers "tossed over the fence" to other conferences (I'm using the conference as an example venue, but I am sure journals, technical press, and bloggers are seeing the same thing). A paper on AI and Systems (or Networks, or Databases, or pick your own long established domain) should bring interesting results in those domains - indeed, it...

Hyperspectrally identifying trees in tropical forests

2024-06-01T01:00:00Z

A preprint on using hyperspectral sensors to perform tree species identification across the tropics is now available on bioarxiv. This study introduces a new approach for mapping tree species linking a multi-temporal implementation of the CNN method detectree2 to segment tree-crowns from aerial photographs to machine learning classifiers to identify species from hyperspectral data. James G. C. Ball , Sadiq Jaffer , Anthony Laybros , Colin Prieur , Toby Jackson , Anil Madhavapeddy , Nicolas Barbier , Gregoire Vincent and David A Coomes . Working paper at bioRxiv . URL (biorxiv.org) DOI BIB PDF

Mon 27 May 2024

2024-05-27T01:00:00Z

Previous: Mon 20 May 2024 Next: Mon 24 Jun 2024 NixOS Modules On Friday we were talking about doing something alternative to Nix, but without a clear statement of the problems with Nix. One clear problem I have with the NixOS ecosytem is the module system. From section 5.1 of the NixOS paper , the module system allows ‘separation of concerns: each module defines a single, “logical” part of the system (e.g., some system service, support for a certain kind of hardware device, or an aspect such as the system path), even though its implementation might cross-cut many “physical” parts (e.g., the derivations that build the boot scripts)’. In one place I can define everything needed for a particular application; I can describe a matrix setup server, reverse proxy, postgres database, dns records, and TLS...

Programming for the Planet

2024-05-27T01:00:00Z

I was invited by Mary Sheeran to deliver a keynoted at Lambda Days , and I decided to go along to talk about my work on Programming for the Planet . The conference was a really vibrant crowd and I would definitely go along in future years. It's best summarised via an interview video they took of all the speakers.

Cross "Border" Digital Infrastructure

2024-05-23T15:07:00Z

So again while at ID4Africa in Cape Town this week, I heard a lot of people talking about Cross Border use of digital identity. Lets talk a bit about infrastructure here, as I'm not sure people are aware of how hard it is to determine, reliably, where a person, or device are located, geograhpically, let alone jurisdictionally. We ( Microsoft Center for Cloud Research ) wrote about this a wwhile back when simply considering the impact of GDPR on Cloud Services and the location of personal data. The infrastructure doesn't tell you where it is - borders are not digital, they are geo-political constructs that only exist in someone's mind. GPS doesn't work in doors, and can be remarkably perverse in cities anyhow. Content providers (e.g. the BBC in the UK) worry about delivery of content (and adverts and charging)...

sustainability of digital wallets for public infrastructure services

2024-05-23T14:52:00Z

One thing occurred to me when listening to people at ID4Africa 24 talk about wallets is that there's a major sustainability problem due specifically to security considerations. Any wallet needs to be trusted if it is used for transactions that involve personal data or money. To implement this trust, the wallet software currently built by major vendors such as Apple, Google and (say) HSBC can use secure enclaves (Trusted Execution ENvironment) support on the device (e.g. trustzone on ARM processors, or variants as built by various handset vendors). However, the supprt varies with time, but with modications to hardware coming along (e.g. future ARM support for multiple realms and attestation) and simply because software and hardware volunerabilities arise, some of the latter being mitgated...

DPG #2 or should I say DPPG or possibly DPPI

2024-05-22T08:06:00Z

We're hearing a lot about DPIs - Digital Public Infrastructure (the Internet, the spectrum for mobile telephone, open banking networking etc).... So then there's a lot of talk about building new Infrastructures for (e.g.) Digital Identity - and provisioning this through Public Private Partnerships - so really we then have a DPPI - indeed, the Internet and WWW and Cloud serve as an example of just that too. But then we have Digital Public Goods - for me, this is an extension of the notion of open source - so again the software that runs the Internet is available in open source form, together with documentation, and even much test data (simulators too). But new systems have evolved new forms of ownership, so a lot of the digital content in the Internet is a mix of open (free) and open...

DPGs #1

2024-05-21T16:27:00Z

The oldest and best example of a digital public good is the Internet. Why people don't start from this is surprising to me: Since 1982, source code of the exemplary implementation from UC Berkeley has been available plus documented in an open access series of books documenting that code and working: TCP/IP Illustrated (vol 2) The key thing here was that every thing accepted as an internet standard had at least 2 interoperating implementations, preferably three, one of which was open source (unencumbred by any IP) - for me, this defines digital (code/data), public (there's no barrier to entry due to ownership restrictive practices) infrastructure (you can run the code and computers are general purppose machines so any computer can run it, subject to resource constraints:-) Two other reference points -...

Mon 20 May 2024

2024-05-20T01:00:00Z

Previous: Mon 13 May 2024 Next: Mon 27 May 2024 Mon: Met with Anil We skimmed the Carbon-aware name resolution paper. Anil said it was a narrow contribution, we’re making too many points in one paper, and that it needs to be framed in a research style with a hypothesis and experimentation attempting to disprove the hypothesis. He liked the mailserver redundancy example (powering up the backup when the primary goes down), which he likened to self-scaling applications managing the power state of machines. He said that the carbon-aware load balancing (Patrick’s section) seems more general than specific to emissions. Anil also said carbon isn’t the best terminology. I agree – we were following HotCarbon’s wording. We talked about doing a simulation to disprove the hypothesis. And how there’s a pareto...

Mon 13 May 2024

2024-05-13T01:00:00Z

Previous: Mon 6 May 2024 Next: Mon 20 May 2024 Wrote ‘Carbon-aware Name Resolution’ with Patrick for HotCarbon but ultimately decided not to submit it as the conference doesn’t seem to align with our interests and the paper could use more polishing. We’re planning on polishing it up and sticking it on arxiv as a basis for a HotNets submission. We spoke to Anil about this and came away with a bunch of ideas, including Managing the power schedule of resource-constrained devices. Re-visiting the idea of giving each service a unique IPv6 address. Store-and-forward energy budgeting for low powered sensor networks. Why isn’t the whole Internet a Filesytem (could the DNS encode file paths?). Providing TLS to MQTT. Having a BGP IPv6 testbed for this with Mythic Beasts. Workload scheduling in a compute...

Mon 6 May 2024

2024-05-06T01:00:00Z

Previous: Mon 29 Apr 2024 Next: Mon 13 May 2024 My Tarides contract is progressing nicely, and we now have Opam setting an environment built from Nix to provide external dependencies. I’ve also indexed all of Nixpkgs history in order to do version solving across it. Spoke to Roy about a possible Part II project, and wrote another proposal on an ‘OCaml Wayland Compositor’; something I’ve been meaning to do for a while but have yet to find the time. Wayland 1 is a communications protocol for next-generation display servers used in Unix-like systems which has been adopted as the default display server by Linux distributions including Fedora with KDE, and Ubuntu and Debian with GNOME. It aims to replace the venerable X display server with a modern alternative. X leaves logic such as window...

Quantifying the impact of the food we eat on species extinctions

2024-05-01T01:00:00Z

Submitted preprint on quantifying the biodiversity cost of global food consumption for peer review Thomas Ball , Michael Dales , Alison Eyres , Jonathan Green , Anil Madhavapeddy , David Williams and Andrew Balmford . Working paper at Cambridge Open Engage . URL (cambridge.org) DOI BIB PDF

Predicting species using machine learning at CCAI

2024-05-01T01:00:00Z

Emily Morris did some great MPhil work here in her Masters on using CNNs with satellite data to do species predictions in South Africa better. She presented it at the ICLR CCAI workshop in Vienna, and is now off to do a PhD at Oxford! Species distribution models are crucial tools that predict species locations by interpolating observed field data with environmental information. We develop an improved, scalable method for species distribution modelling by proposing a dataset pipeline that incorporates global remote sensing imagery, land use classification data, environmental variables, and observation data, and utilising this with CNN models to predict species presence at higher spatial and temporal resolutions than well-established species distribution modelling methods. Emily Morris , Anil Madhavapeddy ,...

Low-latency wayland compositor in OCaml

2024-05-01T01:00:00Z

Low-latency wayland compositor in OCaml This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is currently being worked on by Tom Thorogood . It is co-supervised with Ryan Gibb . When building situated displays and hybrid streaming systems, we need fine-grained composition over what to show on the displays. Wayland is a communications protocol for next-generation display servers used in Unix-like systems. [1] It has been adopted as the default display server by Linux distributions including Fedora with KDE, and Ubuntu and Debian with GNOME. It aims to replace the venerable X display server with a modern alternative. X leaves logic such as window management to application software, which has allowed the proliferation of different approaches. Wayland,...

Diffusion models for terrestrial predictions about land use change

2024-05-01T01:00:00Z

Diffusion models for terrestrial predictions about land use change This is an idea proposed in 2024 as a Cambridge Computer Science Part III or MPhil project, and is available for being worked on. It may be co-supervised with Sadiq Jaffer . This project investigates how to build remote sensing data-driven models for the evolution of landscapes, which we can use to better predict deforestation, flooding and fire risks. Diffusion models are now widespread for image generation and are now being applied to video. [1] In addition the GenCast project from Google Deepmind used a diffusion model ensemble for weather forecasting, resulting in a high degree of accuracy compared to traditional methods. [2] The goal of this project is to train a video diffusion model on time series of...

Mon 29 Apr 2024

2024-04-29T01:00:00Z

Previous: Mon 22 Apr 2024 Next: Mon 6 May 2024 Attended the MirageOS retreat. Spoke to Hannes’ about the DNS library and he asked me to upstream a few things, and talked about my Capnp approach to updates and secondary-ing. Had some productive conversations with Jules which led to an Opam NixOS depext approach Paul-Elliot helped me make my DNS transport library a bit more idiomatically functional, which we presented. Created a gravity-body simulation with Emile using an as-of-yet released OCaml game engine, which we presented. Got some nice window manager and vim tips. Went for a hike in the Atlas mountains. I also had some thoughts on offline operation of networked applications. One of the attendees showed me a video calling application for use in a mesh network that identifies clients by an IP...

Mon 22 Apr 2024

2024-04-22T01:00:00Z

Previous: Mon 15 Apr 2024 Next: Mon 29 Apr 2024 I deployed Eon certificate provisioning to my server to my server, and promptly got rate limited by Let’s Encrypt (staging). It was a combination of not limiting the amount of concurrent certificates being provisioned and a bug in the Cap’N Proto OCaml RPC Eio port that Patrick purportedly has a fix for. I’ve ended up using systemd services extensively to support provisioning certificates. I also added a CalDAV server to Eilean.

Green Urban Equity in the UK

2024-04-19T01:00:00Z

This is part of my presentation at EGU24 in Vienna that was about mapping green spaces in the UK using the 3-30-300 rule. You can find out more about the abstract here .

Mon 15 Apr 2024

2024-04-15T01:00:00Z

Previous: Mon 8 Apr 2024 Next: Mon 22 Apr 2024 I’ve been doing a lot of hacking on Eon to get it to a state where I can use it to provision certificates in production. I’ve written a capnproto schema file that exposes a capability to a domain. When we run a nameserver it outputs a capability for each domain for which it is authoritative. $ cap -z cl.freumh.org --capnp-secret-key-file /var/lib/eon/capnp-secret.pem --capnp-listen-address tcp:cl.freumh.org:7000 --state-dir /var/lib/eon $ sudo ls /var/lib/eon/caps/ cl.freumh.org.cap zone.cap This capability can then be provided to a client. $ capc get-name cl.freumh.org.cap cl.freumh.org The client can create a new capability for a subdomain, which could be passed to a service. NB this is persisted to disk so it can be referenced across reboots. $...

social media convestions and aliens in the ether

2024-04-13T13:00:00Z

Starting locally , I've notice how there are many different conventions about how people use different social media platforms (social networks, email, microblogging, etc) At one extreme, some people DM me on slack - this is annoying as, to save my sanity, i have turned off notifications on everything, and I look at different platforms with different frequencies - slack, mostly, once a day, compared to say, whatsapp (and signal and matrix), once an hour at least. While I don't use. teams for messaging, I know people who do, but they are signed on while at work all working hours, so that works ok for them. At another extreme, some people only use a platform in broadcast mode, so an email list is flooded with "how do I leave this list" messages, or a whatsapp group is flooded with "please...

Mon 8 Apr 2024

2024-04-08T01:00:00Z

Previous: Mon 1 Apr 2024 Next: Mon 15 Apr 2024 Tue some rough working notes for eon’s capability interface to provision TLS certificates: modified the client to read the capability from a file and write the certs to a directory. todo: persistence account key wildcard certs should we do CSR on the client or server? should we do renewals on the client or server? let’s store everything on the server renewals keep the same private key? no, new certificate renewals: studryrefs for capabilities mapped to domains keep track of expiry update interface do we mimic DNS UPDATE? yes do we pass binary blob? no todo provisioning a cert for root https://github.com/mirage/ca-certs-nss https://github.com/suri-framework/castore multiple domains SAN CN extensions Thu done: subject alternative name generate caps for...

Teaching 1A Security and Software Engineering

2024-04-04T01:00:00Z

We're still reeling from the shocking and unexpected passing of Ross Anderson last week. You can read a lovely tribute to him by Satnam Singh , and I still getting my thoughts together on all the guidance, advice and prods in the right direction that Ross has given me over the years. In pragmatic news, I'll be emergency lecturing part of Ross' 1A Software and Security Engineering course here at Cambridge, along with my colleagues Martin, Alastair, Mort and Rob who have all stepped up at short notice. I have absolutely no idea how we'll live up to Ross' standard, but we'll do our very best in his memory!

Global, robust and comparable digital carbon assets

2024-04-04T01:00:00Z

Paper on smart contracts for carbon credits at ICBC 2024 in Dublin Sadiq Jaffer , Michael Dales , Patrick Ferris , Thomas Swinfield , Derek Sorensen , Robin Message , Srinivasan Keshav and Anil Madhavapeddy . Paper in the proceedings of the 6th IEEE International Conference on Blockchain and Cryptocurrency . URL (arxiv.org) DOI BIB PDF

Mon 1 Apr 2024

2024-04-01T01:00:00Z

Previous: Mon 25 Mar 2024 Next: Mon 8 Apr 2024 I read datatracker.ietf.org/doc/draft-many-dnsop-dns-isolated-networks/ which discussed some considerations for DNS in networks with delayed and/or intermittent connectivity. This reminded me of another scenario to consider for Eon : Bootstrap a network in a remote isolated environment. As well as, moving more into the service management territory again, keeping track of service logs.

Rolling out a new site design

2024-04-01T01:00:00Z

I've done a redesign of my site after about 20 years since the last one back in 2003 . The site design is based on my upcoming Bushel content manager, which I'll post about more once I get the data model in place and try it out properly using this site as a guinea pig. Nick Ludlam also refreshed his website since we were chatting about how outdated our web presences were, and he also put up a main recoil.org page for the main server.

Mon 25 Mar 2024

2024-03-25T00:00:00Z

Previous: Mon 18 Mar 2024 Next: Mon 1 Apr 2024 Writing Scientific method: My thesis is that the DNS is the root of the Internet. That is, every network request starts off with a DNS resolution. The DNS is in a great position to manage resources in the Network, like certificates, keys, services, etc. But for some reason people don’t do this now. One advantage Eon has over the state-of-the-art dynamic DNS updates for the ACME DNS challenge is the principle of least privilege. We can provide a capability to only a subdomain, with the thread model a rough service or machine. Regarding scenario 5, the DNS is a very small key value store. “The DNS is a general (if somewhat limited) hierarchical database, and can store almost any kind of data, for almost any purpose,” RFC2181. We can keep track of what...

Mon 18 Mar 2024

2024-03-18T00:00:00Z

Previous: Mon 11 Mar 2024 Next: Mon 25 Mar 2024 Eon Some scenarios we could solve with a client-server API for Eon: We could keep track of the proliferation of federated services in an Eilean deployment including matrix, mastodon, and email. Currently, we need to manually provision DNS records like an A record at mastodon.example.org . We need to run a certbot to respond to a ACME HTTP challenge by configuring a path to serve by a webserver, or respond to a DNS challenge using dynamic DNS UPDATES to update a nameserver. The HTTP challenge pretty heavyweight and doesn’t work so well when e.g. load balancing accross a webserver, and the DNS challenge typically requires providing access to the whole domain. To demultiplex multiple HTTP services on the same machine we typically use a reverse proxy...

Mon 11 Mar 2024

2024-03-11T00:00:00Z

Previous: Mon 26 Feb 2024 Next: Mon 18 Mar 2024 I reworked the EON project structure and Capnp Schema file too: struct Record { name @0 :Text; type @1 :Int32; value @2 :Text; } struct Prereq { union { exists @0 :Record; existsData @1 :Record; notExists @2 :Record; nameInuse @3 :Record; notnameInuse @4 :Record; } } struct Update { union { add @0 :Record; remove @1 :Record; removeAll @2 :Record; removeSingle @3 :Record; } } struct CertReq { # Used to request a certificate for a service union { callback @0 :CertCallback; none @1 :Void; } } interface Zone { # Capability to initalize a Zone for which the nameserver is authorative init @0 (name :Text) -> (domain :Domain); } interface Domain { # Capability for a domain getName @0 () -> (name :Text); # Get the domain name delegate @1 (subdomain :Text) ->...

Witch Consumer Magazine, review of the leader boared top three LLMs "Conformité Ecologique" (the ubiquitous CE marque)

2024-03-10T15:38:00Z

Witch Consumer Magazine, review of the leader boared top three LLMs "Conformité Ecologique" (the ubiquitous CE marque) We analyzed the CE claims of the following three large languish models, with respect to four key metrics for the Ecologique, as agreed in European law, namely enthalpy, internet pollution (measured in LoCS -- libraries of congress), bio-dediversification,and general contribution towards the heat death of the universe. Currently, according to the boared, these are the top-of-the-heap in terms of hype-parameters: The Faux Corperation's Pinocchio Astravista's Libration Sitting Duck's Nine Billion Dogma We hired some prompt engineers to devise a suitably timely benchmark suite, and embedded the the three systems in our whim tunnel taking care to emulate all aspects of the...

Preprint available on insuring against variability of NbS

2024-03-01T00:00:00Z

A new preprint is available on our work on ex-ante pricing models for nature-based solutions. It is currently under review, so any feedback is most welcome! E.-Ping Rau , James Gross , David A Coomes , Thomas Swinfield , Anil Madhavapeddy , Andrew Balmford and Srinivasan Keshav . Journal paper in Carbon Management (vol 15 issue 1). URL (tandfonline.com) DOI BIB PDF

Green Urban Equity: Analyzing the 3-30-300 Rule in UK Cities and Its Socioeconomic Implications

2024-03-01T00:00:00Z

Abstract on urban biodiversity and human health at EGU 24 Andres Zuñiga-Gonzalez , Anil Madhavapeddy and Ronita Bardhan . Technical report (EGU24-20833) at Copernicus Meetings . URL (meetingorganizer.copernicus.org) DOI BIB

A Case for Planetary Computing

2024-03-01T00:00:00Z

Revision of planetary computing preprint Patrick Ferris , Michael Dales , Sadiq Jaffer , Amelia Holcomb , Eleanor Toye Scott , Thomas Swinfield , Alison Eyres , Andrew Balmford , David A Coomes , Srinivasan Keshav and Anil Madhavapeddy . Working paper at arXiv . URL (arxiv.org) DOI BIB

Towards International Goverance of AI

2024-02-26T16:31:00Z

I wonder what people are really thinking when they think of governance of Intelligence? If we were considering human intelligence (which we are by extension) we better tread carefully, especially when considering who owns it. The ability to reason, creatively, to innovate is not really the same as any other thing we have sought governance over - nuclear weapons (test ban treaty, and pugwash convention) spectrum allocation orbits around earth maritime&air traffic - fuels, tracking, control etc recombinant DNA (asilomar conference the weather (and interventions like geo-engineering e.g. see RS report on same ) What's similar about these, and what is different? Well we only have one go at each - there's a very countable human race, planet, sea, zombie apocalypse, climate emergency. we don't have...

Mon 26 Feb 2024

2024-02-26T00:00:00Z

Previous: Mon 19 Feb 2024 Next: Mon 11 Mar 2024 Shark We’ve got an obuilder runc container using Linux network namespaces with VPNkit forwarding outgoing http requests to a cohttp-proxy-lwt. I’ve created a monorepo assembling obuilder, VPNkit, and cohttp and got them building. Some issues were VPNkit having an outdated version of cohttp and having it’s own dune dns library which collides with the Mirage DNS library. Next is assembling a binary to do the container, namespace, VPNkit, and http proxy orchestration. I’m thinking about the UI of what this will look like. Something like $ shark create $ shark attach $ ... We also need to implement some kind of store to cache and replay http queries. Some technical nodes on reproducing this prototype separate-binary setup follow: build...

Mon 19 Feb 2024

2024-02-19T00:00:00Z

Previous: Mon 12 Feb 2024 Next: Mon 26 Feb 2024 Anil Meeting Anil mentioned with QUIC and DNS SVCB records traffic can be UDP-only. We described a service manager using capability DNS with ‘naming’ and ‘livecycle’ at the DNS level. Anil likened this to docker compose service manager for a collection of services. It seems like systemd’s service management plus container and network management. This would allow things like IP-based load balancing, certificate provisioning, Jitsi-spinning up services on request. I need to extend the lenscap to support service management APIs as well as certificate provisioning. OCluster might be a place to look for this. It could do something really simple like just map to systemd. Anil likened it to Tailscale’s new service management dashboard. I brought up...

FOSDEM

2024-02-13T00:00:00Z

Published Tue 13 Feb 2024. Last update Tue 13 Feb 2024. Tags: conferences . I attended the Free and Open source Software Developers’ European Meeting (FOSDEM) in Brussels, Belgium last weekend. There are hundreds of hours of talks in 35 rooms over a period of two days, and rooms are often full to capacity, so it’s impossible to see everything! Thankfully every room is live-streamed and recordings made available after the fact, so you can catch up on anything you miss. Friday On the Eurostar over my travelling companion and I were lamenting about the Nix DSL, and we heard a French accent from behind: Ah, NixOS. See you at FOSDEM then! Saturday The day started with a coffee and a banana (probably not substantial enough in hindsight), an absolutely packed number 71 bus to the ULB Solbosch...

Mon 12 Feb 2024

2024-02-12T00:00:00Z

Previous: Mon 5 Feb 2024 Next: Mon 19 Feb 2024 Capability DNS We have this nameserver that can provision TLS certificates with ACME using the DNS challenge, and expose a Cap’NP proto interface https://github.com/RyanGibb/eon/blob/main/bin/acme/capnp/service_api.capnp . We also have logic to tunnel Cap’NP over DNS https://github.com/RyanGibb/eon/tree/main/lib/transport , and indeed tunnel arbitrary data such as IP-traffic https://github.com/RyanGibb/eon/tree/main/bin/transport/tunnel . Two ideas we had relating to this were exposing a shell interface https://github.com/RyanGibb/eon/tree/main/bin/transport/sod over DNS and tunneling DNS traffic itself over DNS for query privacy a la Oblivious DNS https://github.com/RyanGibb/eon/tree/main/bin/transport/dodo . I’ve had a look for any related work to ‘de-risk’ a paper. The most obvious is Signposts , which performs an ‘effectful’ name resolution to establish a communication channel through features of edge networks that break the end-to-end principle like NATs and proxies. It relies on DNSSEC to secure...

Mon 5 Feb 2024

2024-02-05T00:00:00Z

Previous: Mon 29 Jan 2024 Next: Mon 12 Feb 2024 Shark I spoke to Michael about VanillaOS: It uses OCI images and ABRoot to provide immutability and atomic updates. Iirc it’s similar to Fedora Silverblue in that it’s an ‘immutable’ operating system, but Silverblue uses OST which is a ’git for binaries (keeps track of metadata and I think GCs). This is also used by flatpak. Nix is a bit more general in that is allows two packages to have different versions of a dependency by specifying them be absolute path rather than hard linking like ABRoot/OSTree. NixOS modules make it system configuration version controlled and rollback-able, and you don’t have to deal with ‘configuration updates’ like e.g. Ubuntu prompt you with. The downside is that you need to write your system config in this esoteric DSL. I...

Selected in the AI@CAM challenge for conservation research

2024-02-05T00:00:00Z

We won the AI@CAM challenge that was sent out University wide to find research projects that use AI to tackle society's biggest challenges. Our project on using AI for climate and nature is one of the five selected. The twin climate and biodiversity crises are two of the world’s most complex challenges to tackle. This project aims to develop AI approaches for bringing together a wide range of datasets and accelerating the collation of information. This work will provide up to date, relevant and robust information for researchers and decision-makers working on climate and biodiversity conservation – opening up the possibility for more targeted and effective solutions to some of our world’s most pressing climate and biodiversity challenges. Anil Madhavapeddy , AI-deas challenge co-lead, said: 'Mitigating the impacts...

Uncertainty at scale: how CS hinders climate research

2024-02-01T00:00:00Z

Paper on uncertainty in climate science in Undone CS Patrick Ferris , Michael Dales , Thomas Swinfield , Sadiq Jaffer , Srinivasan Keshav and Anil Madhavapeddy . Working paper at Undone Computer Science . URL (undonecs.sciencesconf.org) BIB PDF

Mon 29 Jan 2024

2024-01-29T00:00:00Z

Previous: Mon 22 Jan 2024 Next: Mon 5 Feb 2024 Interspatial Networking Got a Zigbee2MQTT bridge deployed via NixOS on a Raspberry Pi 5. This required installing an EDK2 raspberry Pi 5 EFI implementation as a bootloader pending U-boot support https://github.com/NixOS/nixpkgs/issues/260754#issuecomment-1908664693 . Had lots of false starts trying to cross-compile aarch64 (and arvm6 for the raspberry pi 1) on x86_64. I got pretty far with aarch64 but I think nixos-install doesn’t support a cross-compiled grub installation. I’m playing around with different ways of exposing this over HTTP, DNS, RPCs, and maybe Matrix. EON To support RPCs over DNS as a bootstapping mechanism for Jess and I’s idea of an identity service I resurrected the DNS tunnelling in EON. There’s a few kinks to work out with payload lengths fitting into domain names, and an odd failure mode...

Mon 22 Jan 2024

2024-01-22T00:00:00Z

Previous: Mon 15 Jan 2024 Next: Mon 29 Jan 2024 It’s been 4 weeks since I updated this log! I fell off the bandwagon after returning from Boston, getting sidetracked adding a capability-based RPC to provision TLS certs with ACME on a nameserver. Interspatial networking: Andrew Moore dumped a whole bunch of Zigbee smart lights and plugs on my desk. I got a Zigbee bridge to MQTT bridge working and hooked fn06.org up to toggle the lights. This is basically me returning to the original vision of my MPhil project! If I can get a website to control the lights there’s no reason an AR headset couldn’t do the same. I’m also envisioning a DNS-based control interface with access control, e.g. dig toggle._lights.fn06.org . If I want to try the Hololens development environment I’ll need to get a GPU windows...

Mon 15 Jan 2024

2024-01-15T00:00:00Z

Previous: Mon 8 Jan 2024 Next: Mon 22 Jan 2024 Figured out how to use ZFS for my NAS and shark.

Mon 8 Jan 2024

2024-01-08T00:00:00Z

Previous: Mon 1 Jan 2024 Next: Mon 15 Jan 2024 Took some time off to see friends and family. Bult a low-power (12W idle) NAS with 32TB storage on NixOS with ZFS. Blog post TODO. I still think there’s something here with using a low-powered board to wake a higher powered computer on e.g. a DNS resolution. Anil mentioned stalwart which might be worth looking at for Eilean.

2023 year in review

2024-01-04T00:00:00Z

A lot has happened in the last year, so I thought it would be good to write up a review. My biggest change in 2023 was that my wife and I had a baby! This has brought a mixture of joys and frustrations, but overall it has been very good. I took three months of full-time parental leave after the birth, and since going back to work I’ve been sharing the parenting with responsibilities with my partner. Family has therefore been my top priority, but I won’t talk much about family things in this post, since I prefer to keep it private. Lots of work things happened as well: New job! As of January 2024 I have a new job as Associate Professor in Cambridge ! Unlike all my previous academic positions, which were all fixed-term contracts of a few years, this is a permanent position. A huge number of people apply for this...

Tiny Code Christmas - part 2: learning OCaml and that demos are mostly functional

2024-01-03T09:57:06Z

This post is a follow on to the previous part 1 about Tiny Code Christmas 2022 , and is about what I did for TCC 2023. In order to get more info on what TCC is and why it's a fun thing to do, I recommend you start there, but if you're just interested in me messing around with OCaml , then you're in the right place. There were two things that motivated me to tackle this years TCC in the OCaml language. Firstly, a bunch of my colleagues at work use OCaml, and indeed work on language, and so there's been a bunch of encouragement from them that I should join in. However, I find I have two modes when it comes to working with programming languages: I can either work on a problem I'm not familiar with in a language I'm comfortable with, or I can do the inverse and tackle problems that I'm comfortable with in a...

Tiny Code Christmas - part 1: making coding fun through constraints and community

2024-01-02T13:00:56Z

Normally, as someone who codes for a living, xmas is a time for me to down tools and step away from the computer for a bit, spent time with family, spend some time in the workshop and generally not stare at a computer so much. But this last two years I've had a lot of fun doing some small amounts of coding for fun. In this and the next post, I'm going to go through what got me doing this, what I learned from it, and why I recommend it to others. But the TL;DR is along the lines of: small regular constrained challenges, community, and exploration. Your browser does not support the video element. In an post I wrote a couple of years ago I mentioned in passing the idea of fantasy consoles like TIC80 : these are programs that pretend to emulate a retro computer from the 8 or 16 bit era of...

Mon 1 Jan 2024

2024-01-01T00:00:00Z

Previous: Mon 25 Dec 2023 Next: Mon 8 Jan 2024 Set up a local plaintext email setup with mbsync/goimapnotify/aerc/mu/msmtp/gpgp/ldapsearch nix .

Spatial and multi-modal extraction from conservation literature

2024-01-01T00:00:00Z

Spatial and multi-modal extraction from conservation literature This is an idea proposed in 2024 as a Cambridge Computer Science Part III or MPhil project, and is available for being worked on. It may be co-supervised with Sadiq Jaffer , Alec Christie and Bill Sutherland . The Conservation Evidence Copilots database contains information on numerous conservation actions and their supporting evidence. We also have access to a large corpus of academic literature detailing species presence and threats which we have assembled in Cambridge in collaboration with the various journal publishers. This MPhil project aims to combine these published literature resources with geographic information to propose conservation interventions. The goal is to identify actions that are likely to be effective based on prior evidence and have...

Privacy preserving emissions disclosure techniques

2024-01-01T00:00:00Z

Privacy preserving emissions disclosure techniques This is an idea proposed in 2024 as a Cambridge Computer Science PhD topic, and is currently being worked on by Jessica Man . It is co-supervised with Martin Kleppmann . Customers of online services may want to take carbon emissions into account when deciding which service to use, but are currently hindered by a lack of reliable emissions data that is comparable across services. Calculating accurate carbon emissions across a cloud computing pipeline involves a number of stakeholders, none of whom are incentivised to accurately report their emissions for competitive reasons. This PhD explores mechanisms to support verifiable and privacy-preserving emissions reporting across a chain of energy suppliers, cloud data centres, virtual machine hosting...

Low-power sensing infrastructure for biodiversity

2024-01-01T00:00:00Z

Low-power sensing infrastructure for biodiversity This is an idea proposed in 2024 as a Cambridge Computer Science PhD topic, and is currently being worked on by Josh Millar . It is co-supervised with Hamed Haddadi . In-situ sensing devices need to be deployed in remote environments for long periods of time, and minimizing their power consumption is vital for maximising both their operational lifetime and coverage. We are exploring the construction of a versatile multi-sensor device (initially based around the ESP32 chipset) and designing an exceptionally low power consumption model by using an on-device reinforcement learning scheduler that can learn to cooperate with other nearby devices. Our prototype device setup for learning schedules for biodiversity monitoring does pretty well against...

Legal perspectives on integrity issues in forest carbon

2024-01-01T00:00:00Z

Legal perspectives on integrity issues in forest carbon This is an idea proposed in 2024 as a postdoctoral project, and has been completed by Sophie Chapman . It was co-supervised with Eleanor Toye Scott . Carbon finance offers a vital way to fund urgently needed forest conservation, but there are integrity issues on the supply side. [1] Besides the known issues with carbon quantification, [2] carbon credits are often poorly designed and implemented from a legal perspective. Specifically, in the absence of a clear legal framework for forest carbon credits, contracts tend to conceptualise credits in similar terms to the products of extractive industries, such as mineral mining. This is a factually inaccurate model for carbon credits, since the carbon is not extracted but on the contrary...

Generating chunk-free embeddings for LLMs

2024-01-01T00:00:00Z

Generating chunk-free embeddings for LLMs This is an idea proposed in 2024 as a Cambridge Computer Science Part III or MPhil project, and is currently being worked on by Mark Jacobsen . It is co-supervised with Sadiq Jaffer . This project aims to explore the development of a chunk-free approach for generating embeddings in Retrieval-Augmented Generation (RAG) models. Traditional RAG workflows often involve manual or predefined chunking of documents, and we seek to bypass this requirement. Instead, our approach involves generating multiple embeddings for unchunked text using a synthetic dataset created by (e.g.) a 7b parameter LLM. This dataset would feature structured, point-by-point summaries of each paragraph. An off-the-shelf embedding model could then be modified by removing its mean...

Evaluating RAG pipelines for conservation evidence

2024-01-01T00:00:00Z

Evaluating RAG pipelines for conservation evidence This is an idea proposed in 2024 as a good starter project, and has been completed by Radhika Iyer . It was co-supervised with Sadiq Jaffer . At the Conservation Evidence Copilots project, we are interested in finding and synthesising evidence for conservation interventions. Once we have this evidence, it needs to be synthesised into a form that can be used to inform policy and practice via natural language query interfaces. One way to do this is to use a RAG (Retrieval Augmented Generation) pipeline, which can automatically retrieve relevant information from a large corpus of documents, analyse it to extract key information relevant to CE, and generate a summary of the key findings. This project involved involve evaluating the performance of RAG pipelines for conservation evidence,...

Effective geospatial code in OCaml

2024-01-01T00:00:00Z

Effective geospatial code in OCaml This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is currently being worked on by George Pool . It is co-supervised with Michael Dales and Patrick Ferris . Geospatial data processing is a critical component of many scientific and engineering workflows, from environmental monitoring to urban planning. However, writing geospatial code that scales to multiple cores and makes best use of available memory can be challenging due to the scale of the data involved. To deal with this, we have been developing some domain-specific tools to improve the state of affairs. Yirgacheffe is a wrapper to the GDAL library that provides high-level Python APIs that take care of figuring out if datasets overlap, and if vector layers need to be...

Deep learning for decomposing sound into vector audio

2024-01-01T00:00:00Z

Deep learning for decomposing sound into vector audio This is an idea proposed in 2024 as a Cambridge Computer Science Part III or MPhil project, and is available for being worked on. It may be co-supervised with Trevor Agus . All that we hear is mediated through cues transmitted to the brain from the cochlea, which acts like a bank of auditory filters centred at a wide range of centre frequencies. A lot of our knowledge of hearing comes from psychoacoustical experiments that involve simple sounds, like sine waves, whose synthesis parameters are closely related to cues available beyond the cochlea. However, for recorded sounds, many types of cue are available, but our use of these cues is limited by the extent that these cues can be manipulated in a controlled fashion. [1]...

Crawling grey literature for conservation evidence

2024-01-01T00:00:00Z

Crawling grey literature for conservation evidence This is an idea proposed in 2024 as a good starter project, and has been completed by Shrey Biswas and Kacper Michalik . It was co-supervised with Sadiq Jaffer . At the Conservation Evidence Copilots project, we are interested in finding and synthesising evidence for conservation interventions. Much of this evidence is published in academic journals, but there is a large body of grey literature that is not indexed in academic databases. This includes reports from NGOs, government agencies, and other organisations that are not peer-reviewed, but can still contain valuable information. This project involved developing a web crawler to search for grey literature on conservation interventions, tracking the provenance and license information, and extracting relevant...

Conservation Evidence Copilots

2024-01-01T00:00:00Z

Conservation Evidence Copilots The Conservation Evidence team at the University of Cambridge has spent years screening 1.6m+ scientific papers on conservation, as well as manually summarising 8600+ studies relating to conservation actions. However, progress is limited by the specialised skills needed to screen and summarise relevant studies -- it took more than 75 person years to manually curate the current database and only a few 100 papers can be added each year! We are working on AI-driven techniques to accelerate addition of robust evidence to the CE database via automated literature scanning, LLM-based copilots and scanning of grey literature. We aim to provide co-pilots that augment human decision making to figure out how to categorise interventions much more quickly and accurately, and ultimately...

Composable diffing for heterogenous file formats

2024-01-01T00:00:00Z

Composable diffing for heterogenous file formats This is an idea proposed in 2024 as a Cambridge Computer Science Part III or MPhil project, and is available for being worked on. It may be co-supervised with Patrick Ferris . When dealing with large scale geospatial data, we also have to deal with a variety of file formats, such as CSV, JSON, GeoJSON, or GeoTIFFs, etc. Each of these file formats has its own structure and semantics, and it is often necessary to compare and merge data across different file formats. The conventional solution with source code would be to use a tool such as Git to compare and merge data across different file formats. However, this approach is not always feasible, as it requires the data to be in a text-based format and the data to be structured in a way that can be...

Building bigraphs of the real world

2024-01-01T00:00:00Z

Building bigraphs of the real world This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is currently being worked on by Roy Ang . It is co-supervised with Ryan Gibb . Bigraphs were originally proposed as a model for the behaviour of ubiquitous systems since interaction between mobile devices is dependent on both placing (locality) and linking (connectivity). However, there has yet to be a bigraph that represents the complete physical world. Such a bigraph will enhance the computer's representation of its location from a simple latitude-longitude pair to a context more familiar to humans: the room it is in, the street the building is on, and the town the street is in. This will allow for location-aware applications and policies about connectivity of...

Assessing mangrove literature for conservation evidence

2024-01-01T00:00:00Z

Assessing mangrove literature for conservation evidence This is an idea proposed in 2024 as a Cambridge Computer Science Part II project, and is available for being worked on. It may be co-supervised with Sadiq Jaffer and Tom Worthington . At the Conservation Evidence Copilots project, we are interested in finding and synthesising evidence for conservation interventions. Mangrove forests are one of the most threatened ecosystems in the world, and there is a large body of literature on the topic. However, this literature is spread across a wide range of sources, including academic journals, reports, and grey literature, and it can be difficult to identify and synthesise the key findings. Moreover, there is a need to assess the quality of the evidence, perhaps from remote sensing data or field studies, to inform...

Accurate summarisation of threats for conservation evidence literature

2024-01-01T00:00:00Z

Accurate summarisation of threats for conservation evidence literature This is an idea proposed in 2024 as a Cambridge Computer Science Part III or MPhil project, and is currently being worked on by Kittson Hamill . It is co-supervised with Sadiq Jaffer . At the Conservation Evidence Copilots project, we are interested in constructing a taxonomy of threats to wildlife from the literature. This involves scanning the body of conservation literature and gathering/synthesising evidence for conservation interventions from a threats perspective. Once the text has been retrieved, it needs to be summarised in a way that is accurate, concise and relevant and verified with human experts. This is particularly important for conservation evidence, where the key findings need to be communicated clearly to inform policy and...

Mon 25 Dec 2023

2023-12-25T00:00:00Z

Previous: Mon 18 Dec 2023 Next: Mon 1 Jan 2024 Added Let’s Encrypt Nameserver Daemon ( lend ) and client ( lenc ) to eon, which provisions TLS certs with the ACME DNS challenge inside the nameserver modifying the trie directly. Required porting ocaml/letsencrypt to Eio 0.12. Added a Capnp RPC interface to provision certificates using this creating Let’s Encrypt Nameserver Cap’n Proto ( lenscap ). There’s still work to do including supporting renewals with capnp callbacks which will probably require sturdy refs, the story for which persisting seems complicated. Read RFC9518 on the train up to Scotland. It talks about how technical decentralisation is necessary but not sufficient. Section 3.1 talks about issues with federation in the centralisation of SMTP providers and how XMPP federation is voluntary. Matrix bridges...

Mon 18 Dec 2023

2023-12-18T00:00:00Z

Previous: Mon 11 Dec 2023 Next: Mon 25 Dec 2023 Majorly refactored eon including porting to Eio 0.12 moving from objects to polymorphic varients.

NixOS channels

2023-12-13T00:00:00Z

I don’t pretend to understand NixOS configuration fully yet, what with Flakes and channels and so forth. But I did find the following usful to setup channels consistently so that I could have a single config that used both. sudo nix-channel -- list # to list known channels sudo nix-channel -- remove nixos # to remove a channel sudo nix-channel -- add https://nixos.org/channels/nixos-unstable nixos # to go bleeding edge …ultimately resulting in: $ nix-channel -- list # channels are stored per-user, and $(whoami) != root $ sudo nix-channel -- list nixos https://nixos.org/channels/nixos-unstable nixpkgs https://nixos.org/channels/nixos-unstable Upgrading to the latest release is then something like: pushd ~ /rc-files/nixos/ sudo nix-channel -- update nix flake update sudo nixos-rebuild...

Mon 11 Dec 2023

2023-12-11T00:00:00Z

Previous: Mon 4 Dec 2023 Next: Mon 18 Dec 2023 Report for Huwawi Studentship Update Sent to Ben Karniely. My research over the past year has focused on addressing a fundamental limitation in the current Internet architecture - the lack of support for naming physical locations and resolving them to various addressing mechanisms beyond IP. To address this gap, we have proposed the Spatial Name System (SNS), which enables the assignment of hierarchical location-based names and provides global and local resolution schemes. The SNS extends the Domain Name System (DNS), allowing the integration of spatial names into existing applications and opening up new possibilities for sensor networks and augmented reality. Thanks to Huawei’s Hisilicon Studentship I had the opportunity to present this research at...

Mon 4 Dec 2023

2023-12-04T00:00:00Z

Previous: Mon 27 Nov 2023 Next: Mon 11 Dec 2023 Arrived in Cambridge and finialised my HotNets presentation adding some new slides to expand on how existing DNS mechanisms like DNS-SD can interoperate, illustrating the interface to a geodetic query, etc. Attended the HotNets Conference. Met with Jane Street. HotNets Conference It was great! Feedback on the SNS First and foremost, I recieved some really useful questions, thoughts, and feedback on the SNS. While presenting I was asked: Joshua Smith, Uni of Washington: Q: does this support mobile device? A: that is not our primary use case with these device with their functional primarily defined by their location, though a limitted form of mobility could be supported through existing mechanisms like CNAMEs. Q: what about devices that are inbetween…...

Joined the Royal Society fellowships committee

2023-12-02T00:00:00Z

I joined the Royal Society Newton International Fellowships committee to help with selecting bright new scientists from abroad who wish to conduct research in the UK. The Newton International Fellowship (NIF) programme provides support for outstanding early career researchers to make a first step towards developing an independent research career through gaining experience across international borders. The fellowships enable researchers to access expertise, gain new perspectives and build long-lasting collaborative relationships. -- The Royal Society

PACT Tropical Moist Forest Accreditation Methodology

2023-12-01T00:00:00Z

We have just released the Tropical Moist Forest v2.0 specification, to update the v1.1 released earlier in the year. There are significant updates to the methodology to better match the scheme described in Realizing the social value of impermanent carbon credits . Andrew Balmford , David A Coomes , Michael Dales , Patrick Ferris , James Hartup , Sadiq Jaffer , Srinivasan Keshav , Miranda Lam , Anil Madhavapeddy , Robin Message , E.-Ping Rau , Thomas Swinfield , Charlotte Wheeler and Abby Williams . Working paper at Cambridge Open Engage . URL (cambridge.org) DOI BIB PDF

Mon 27 Nov 2023

2023-11-27T00:00:00Z

Previous: Mon 20 Nov 2023 Next: Mon 4 Dec 2023 Sorted HotNets travel, accomadation, insurance, etc. Travelled to Boston. Recovered from jetlag and had thanksgiving with my American family. Visited Newton, Ludlow, Springfield, North Hampton, Shirley, Boston, and Cambridge. Spent a couple of days preparing for my presentation. Wrote a 2000 word script that I cut to about 1500 words. Ran through presenting it 3 or 4 times, timing it to make sure I wasn’t overrunning.

Mon 20 Nov 2023

2023-11-20T00:00:00Z

Previous: Mon 13 Nov 2023 Next: Mon 27 Nov 2023 HotNets I sent an email out to the SRG mailing list asking if anyone knew anyone at MIT. Jörg Ott said I “may wanna talk to Karen Sollins and John Wroclawski”. Malte said he will be there, and can introduce me to “some of the MIT people who will likely attend from [his] time in PDOS”.

Mon 13 Nov 2023

2023-11-13T00:00:00Z

Previous: Mon 6 Nov 2023 Next: Mon 20 Nov 2023 New Collaborators? Justas Justas suggested collaborating on something with the Adaptive City project sensors. Grenoble Andrzej Duda and Simon Fernandez from Grenoble INP Ensimag LIG Lab are working on very similar things to the SNS. They reached out to Jon and are keen to collaborate. HotNets I can’t seem to find PDFs for a lot of the HotNets papers. The ACM proceedings doesn’t seem to have PDFs, neither does the conference website (and the publishing chair has not got back to me about the availablility on the website like in previous years), so if the author hasn’t put it up on their personal website it’s not available. Teaching Marked and supervised the last concurrent exercise sheet for Pembroke part IBs. We’ve done 4 supervisions for...

Mon 6 Nov 2023

2023-11-06T00:00:00Z

Previous: Mon 30 Oct 2023 Next: Mon 13 Nov 2023 EEG Matrix Server Following up on Matrix Shibboleth integration I deployed an EEG Matrix server. Configure synapse for saml2 https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#saml2_config Had some issues with synapse detecting the pysaml2 module… my personal nixos nixpkgs revision seemed to work. Cambridge raven identity provider (IdP) attributes: https://wiki.cam.ac.uk/raven/Attributes_released_by_the_Raven_IdP How to configure pysaml2 attribute mapping (to map to those synapse expects ) https://pysaml2.readthedocs.io/en/latest/howto/config.html#attribute-map-dir Register our service provider (SP) with the university https://wiki.cam.ac.uk/raven/SP_registration at https://metadata.raven.cam.ac.uk/ . Got the site XML from the path /_matrix/saml2/metadata.xml . In future, maybe we’d want to register with the UK federation. Got the remote url from https://docs.raven.cam.ac.uk/en/latest/reference-apache-saml2/ Got the flu Unfortunately, the rest of the week was a write-off with the flu.

BBC report on the new Cambridge supercomputer ("Dawn") announced at the 2023 AI Summit

2023-11-02T00:00:00Z

On the BBC briefly about the Dawn supercomputer

More on Icons

2023-11-01T19:12:03Z

This is a sort of follow on to a post I wrote about how icons becoming homogenous and hard to distinguish , and inspired by this image posted to social media by Michael Flarup showing the evolution of the macOS default dock over many versions: Whilst I'd be the first to admit to not being a fan of the heavily skeuomorphic apps like the old Notes.app with its faux leather titlebar and the old game center app, I also think that the older icons for notes and such were much more distinctive when you could recognise items by external shape as well as the content. This reminder of what we've lost in terms of both usability and character in our icons inspired me to go back to this vibe with my placeholder icon for a little desktop app I've been writing for myself. The app has the working title of "BAM", and so a literal explosion of shape and colour...

Where on Earth is the Spatial Name System?

2023-11-01T00:00:00Z

Paper on spatial networks on DNS at HotNets 2023 Ryan Gibb , Anil Madhavapeddy and Jon Crowcroft . Paper in the proceedings of the 22nd ACM Workshop on Hot Topics in Networks . URL (dl.acm.org) DOI BIB PDF

Nature Climate Change paper on impermanent carbon credits

2023-11-01T00:00:00Z

Our paper on valuing impermanent carbon credits has been published at Nature Climate Change . It has received a bunch of press coverage, including phys.org , cam.ac.uk , and Mirage . Andrew Balmford , Srinivasan Keshav , Frank Venmans , David A Coomes , Ben Groom , Anil Madhavapeddy and Thomas Swinfield . Journal paper in Nature Climate Change (vol 13 issue 11). URL (nature.com) DOI BIB PDF

First preprint of LIFE biodiversity metric available

2023-11-01T00:00:00Z

The first preprint on our new LIFE metric for global biodiversity is now available. It is under review, so feedback would be very welcome. Alison Eyres , Thomas Ball , Michael Dales , Thomas Swinfield , Andy Arnell , Daniele Baisero , América Paz Durán , Jonathan Green , Rhys Green , Anil Madhavapeddy and Andrew Balmford . Journal paper in Philosophical Transactions of the Royal Society (vol 380 issue 1917). URL (royalsocietypublishing.org) DOI BIB PDF

Mon 30 Oct 2023

2023-10-30T00:00:00Z

Previous: Mon 23 Oct 2023 Next: Mon 6 Nov 2023 HotNets Organised travel dates, travel, and accommodation for HotNets. I was awarded a Queens’ College Perry Travel Award to the value of £740! Met with Anil Agreed to schedule dates to visit NYC and then find a time to meet Jane Street to discuss the polyglot package management project proposal. I’m going to take the train from Boston to NYC! Anil said I’ve got to get a Junior Ranger Badge from a National Park in NYC . Anil recommended live blogging the conference on Mastodon. Talked about what a HotNets presentation could look like, and whether a live demo was feasible. Maybe I could pull AEON and Patrick’s OCaml R-Tree together into something workable. I think while very cool the APIs just aren’t there for the AR demo on a hololens. A smartphone (e.g. iPhone LIDAR)...

Mon 23 Oct 2023

2023-10-23T01:00:00Z

Previous: Mon 16 Oct 2023 Next: Mon 30 Oct 2023 HotNets, HotNets, HotNets Don’t try and rewrite your whole paper a day before the deadline. Got it in, and need to think about travel and accomadation next. The program has been published, and I’m on the first day at 1:15pm - 2:45pm in ‘Session 3: The architecture of the Internet, and datacenter networks’. SIGCOMM SNS Magnus and I threw some ideas around on Thurday morning. Apple’s ARKit might be something to look at in lieu of a Vision Pro. We also discussed getting some experience with iPhone’s lidar precision, how to bootstrap a connection, the issues with civic name encodings, and the necessity to get some estimation of load. Regarding an implementation, once we have a bit more of an idea of what it might look like, github.com/RyanGibb/aeon would be a good...

Cambridge Zero highlights University efforts at Climate Week NYC

2023-10-18T01:00:00Z

I was on stage in New York for Mission Possible during NYC Climate Week . I was there with Emily Shuckburgh and we met with a lot of Cambridge alumni who are all engaged with climate change related activities -- either directly in their careers, or through a side interest. The major highlights on the discussions with alumni centred around agency: a lot of them were wondering how to combine the evidence coming out Cambridge research and combine it with real policy action. A number of the alumni are obviously highly successful in their individual careers, and so the University helping to glue this together would potentially result in valuable actions that might not otherwise come together. This reminded me strongly of the discussions we had in Pembroke a little while back when Emily Shuckburgh chaired my talk...

Mon 16 Oct 2023

2023-10-16T01:00:00Z

Previous: Mon 9 Oct 2023 Next: Mon 23 Oct 2023 HotNets Converted the SNS paper to the ACM final format. Spent way to long figuring out how to make the paper open access. While trying to figure this out I spoke to Jon, Chris, Ardwin, Justas, Sadiq, Helen, and Mort. Mort explained the difference between the copyright, licensing publishing rights, and open access. In the end, we had to use our institutional email addresses to have the ACM’s open access policy kick in. Addressed Magnus’ comments on the paper, and fixed some citations. Pulled the discussion into the introduction. Hibernia Had an idea regarding the energy saving ‘wake-up’ device: the network infrastructure has to be running anyway, so why not push this functionality into the router, which are mostly running Linux now anyway. We...

Mon 9 Oct 2023

2023-10-09T01:00:00Z

Next: Mon 16 Oct 2023 After trying to use Obsidian, a remarkable E-ink tablet, Logseq, emacs org-mode, and plaintext files, I’m using a paper notebook research log for the upcoming academic year and hopefully beyond. I’ve tried this before without much success, but this time I plan to use it more thoughtfully. I’m also trying something new in typing up my weekly notes in a web log. So last week… Eilean I’ve had a couple of requests to get Eilean into a useable state, so I’ve tested (and fixed a bunch of bugs in) it’s deployment on a fresh VM from the NixOS template provided, and added a comprehensive getting started document. Work to be done includes using ACME DNS-01 challenges to provision TLS certificates and automatically incrementing the DNS SOA serial number. Having a modular and...

Information Flow Tracking for Heterogeneous Compartmentalized Software

2023-10-01T01:00:00Z

Paper on DIFC Deluminator interface at RAID 2023 Zahra Tarkhani and Anil Madhavapeddy . Paper in the proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses . URL (dl.acm.org) DOI BIB PDF

Enabling Lightweight Privilege Separation in Applications with MicroGuards

2023-10-01T01:00:00Z

Paper on microgrounds memory API at ACNSW Zahra Tarkhani and Anil Madhavapeddy . Paper in the applied Cryptography and Network Security Workshops . URL (link.springer.com) DOI BIB PDF

Functional Programming for the Planet

2023-09-05T01:00:00Z

Keynoted at ICFP 2023 on Functional Programming for the Planet

State of the OCaml Platform 2023

2023-09-01T01:00:00Z

We deliver the annual presentation about the OCaml Platform in the OCaml Workshop at ICFP 2023. This paper reflects on a decade of progress and developments within the OCaml Platform, from its inception in 2013 with the release of opam 1.0, to today where it stands as a robust toolchain for OCaml developers. We review the last three years in detail, emphasizing the advancements and innovations that have shaped the OCaml development landscape and highlighting key milestones such as the migration to Dune as the primary build system, and the development of a Language Server Protocol (LSP) server for OCaml. Thibaut Mattio , Anil Madhavapeddy , Thomas Gazagnaire and David Allsopp . Paper in the proceedings of the 2023 OCaml Users and Developers Workshop . URL (icfp23.sigplan.org) BIB PDF

Scheduling for Reduced Tail Latencies in Highly Utilised Datacenters

2023-09-01T01:00:00Z

Scheduling for Reduced Tail Latencies in Highly Utilised Datacenters This is an idea proposed in 2023 as a Cambridge Computer Science PhD topic, and has been completed by Smita Vijayakumar . It was co-supervised with Evangelia Kalyvianaki . Modern datacenters have become the backbone for running diverse workloads that increas- ingly comprise data-parallel computational jobs. Due to the ease of use and diversity of resources they host there has been an exponential rise in the demand for datacenters leading to high volume of traffic. Datacenters execute thousands of jobs by scheduling billions of tasks every day. To meet these demands, datacenters providers operate their clusters at levels of high utilisation. We show that under such conditions existing scheduling designs impose large wait times on tail...

Eio 1.0 – Effects-based IO for OCaml 5

2023-09-01T01:00:00Z

An update on the OCaml EIO library at the OCaml Workshop 2023 Thomas Leonard , Patrick Ferris , Christiano Haesbaert , Lucas Pluvinage , Vesa Karvonen , Sudha Parimala , KC Sivaramakrishnan , Vincent Balat and Anil Madhavapeddy . Paper in the proceedings of the 2023 OCaml Users and Developers Workshop . URL (icfp23.sigplan.org) BIB PDF

OneDrive and NixOS

2023-08-22T01:00:00Z

https://github.com/abraunegg/onedrive/blob/master/docs/SharePoint-Shared-Libraries.md https://github.com/NixOS/nixpkgs/pull/77734#issuecomment-575874225 FWIW I also use that package on a NixOS system (via a Nix package that can presumably be installed on other systems if you add nix as a package manager), https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNixOS%2Fnixpkgs%2Fblob%2Fmaster%2Fnixos%2Fmodules%2Fservices%2Fnetworking%2Fonedrive.nix&data=05%7C02%7Crmm1002%40universityofcambridgecloud.onmicrosoft.com%7C8792d760606b4a3e80d108dcbd47b53a%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638593362895323916%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=qPJyO0hTLRnrkbssZ0Y4cLfhaygDiEQtlQ0pwAjgMwM%3D&r...

Pandas vs Efficiency

2023-08-07T11:21:45Z

As part of my role at the Cambridge Center for Carbon Credits (4C) , working closely with ecologists in processing of data, I've become an accidental data scientist, by which I mean instead of loading CSV files directly I now use pandas . Pandas is a popular library that makes working with large arrays of data easier. Unlike numpy , which is there to make it easy to process data in multidimensional arrays, pandas works one abstraction higher, giving columns names and treating the data as tabular, rather than just raw numbers. But similar to numpy, it provides nice syntax to help you work with that data without worrying about individual elements, you tend to work at that tabular level, and that makes it particularly powerful for someone who isn't a native programmer, rather someone who is a domain expert in...

Why using Slack is bad for GeoTIFF Rasters

2023-08-03T14:48:18Z

In the past I've written about how GeoTIFF is one of the most common datas formats used for geospatial data. A GeoTIFF is just a standard TIFF image with a few extra fields on it that mean geospatial tools such as QGIS or libraries such as GDAL know what geographic region this data is referring to, and what map projection system its stored relative to. For example, here is a GeoTIFF I generated showing me where in Sweden you might find moose (meese?): Perhaps I'm sufficiently excited by the notion of where the moose can be found that I want to share this with one of my colleagues via our workplace Slack channel. So I drop the file in, they download it, but then they complain that the image is now all wrong: If we zoom out a bit, we find not only does the image look funny, it's...

Meta Properties of Financial Smart Contracts

2023-08-01T01:00:00Z

Meta Properties of Financial Smart Contracts This is an idea proposed in 2023 as a Cambridge Computer Science PhD topic, and has been completed by Derek Sorensen . It was co-supervised with Srinivasan Keshav . Financial smart contracts routinely manage billions of US dollars worth of digital assets, making bugs in smart contracts extremely costly, and are also increasingly being used in other areas of endeavour such as carbon credit tracking. Because of this, much work has been done in formal verification of smart contracts to prove a contract correct with regards to its specification. However, financial smart contracts have complicated specifications, and it is not all straightforward for humans to write one which correctly captures all of its intended high-level behaviors. To mitigate this...

Computational Models for Scientific Exploration

2023-08-01T01:00:00Z

Computational Models for Scientific Exploration This is an idea proposed in 2023 as a Cambridge Computer Science PhD topic, and is currently being worked on by Patrick Ferris . It is co-supervised with Srinivasan Keshav . The modern scientific method has become highly computational, but computer science hasn't entirely caught up and is sometimes hindering research progress. We use climate science and ecology computation needs as a case study, we are conducting a systematic study in the sources of uncertainty in these fields. We are also designing and implementing a specification language and hermetic computation environment that empowers climate scientists and ecologists to create less ambiguous, more precise and testable scientific methodologies and results, while preserving the ability to explore...

Preprint on the social value of impermanent carbon credits

2023-07-01T01:00:00Z

We have uploaded a preprint of our 4C paper on valuing impermanent carbon credits, by using the Social Cost of Carbon as a basis for a discount function into the future. Comments and feedback are most welcome. Andrew Balmford , Srinivasan Keshav , Frank Venmans , David A Coomes , Ben Groom , Anil Madhavapeddy and Thomas Swinfield . Journal paper in Nature Climate Change (vol 13 issue 11). URL (nature.com) DOI BIB PDF

OCaml wins the ACM Programming Language Software award

2023-06-19T01:00:00Z

I was honoured to be included in the OCaml team that won the ACM Programming Languages Software Award for 2023 . The Association for Computing Machinery (ACM), the world's largest association of computing professionals, today gave the 2023 SIGPLAN Award to a group of developers for their work on the functional programming language OCaml. The award was presented at the annual SIGPLAN Programming Language Design and Implementation Conference to a group of researchers and developers including our colleague Anil Madhavapeddy, Professor of Planetary Computing here. The prestigious Programming Languages Software Award is given annually "to an institution or individual(s) to recognise the development of a software system that has had a significant impact on programming language research, implementations, and tools," ACM says. --...

Developing OCaml with Nix

2023-06-07T01:00:00Z

Published Wed 7 Jun 2023. Last update Wed 7 Jun 2023. Lately, I’ve been writing a significant amount of OCaml as part of my PhD. Instead of using the OCaml package manager (opam) command-line interface (CLI) for these projects, I prefer to use Nix to provide declarative and reproducible development environments and builds. However I still want to be able to interoperate with opam’s file format and access packages from the opam repository. In this blog post we’ll walk through creating a flake.nix file to do this for a hello world project at github.com/RyanGibb/ocaml-nix-hello . Our aim is to make building an OCaml project, and setting up a development environment, as simple as one command. Nix? I’ve said that Nix can provide declarative and reproducible environments and builds. Let’s break down what this...

PACT Tropical Moist Forest Accreditation Methodology

2023-06-01T01:00:00Z

We have just published the Tropical Moist Forest v1.0 specification, which is a detailed description of the methodology we are using for counterfactual dynamic baselines to calculate the additionality, leakage and permanence behind REDD+ projects. I explained some of the background behind this in a seminar last year. Andrew Balmford , David A Coomes , Michael Dales , Patrick Ferris , James Hartup , Sadiq Jaffer , Srinivasan Keshav , Miranda Lam , Anil Madhavapeddy , Robin Message , E.-Ping Rau , Thomas Swinfield , Charlotte Wheeler and Abby Williams . Working paper at Cambridge Open Engage . URL (cambridge.org) DOI BIB PDF

Build your own Stacks Project in 10 minutes

2023-05-14T01:00:00Z

The Stacks project is the most successful scientific hypertext project in history. Its goal is to lay the foundations for the theory of algebraic stacks; to facilitate its scalable and sustainable development, several important innovations have been introduced, with the tags system being the most striking. Each tag refers to a unique item (section, lemma, theorem, etc.) in order for this project to be referenceable. These tags don't change even if the item moves within the text. ( Tags explained , The Stacks Project ). Many working scientists, students, and hobbyists have wished to create their own tag-based hypertext knowledge base, but the combination of tools historically required to make this happen are extremely daunting. Both the Stacks project and Kerodon use a cluster of software called Gerby ,...

Credit credibility threatens forests

2023-05-01T01:00:00Z

Our perspective in Science magazine appeared this week on the credibility of carbon credits and its importance for tropical forest protection. Addressing global warming requires increased investment in conserving and restoring carbon-dense natural habitats. Some companies that emit carbon have turned to certified carbon credits to offset their environmental impact. However, the effectiveness of carbon credits depends on the methods used to quantify them. If carbon credits do not accurately represent their environmental benefits, relying on them could exacerbate climate change. To ensure that carbon credits are robust, the methods used to calculate them must be improved. Andrew Balmford , Pedro H. S. Brancalion , David A Coomes , Ben Filewod , Ben Groom , Alejandro Guizar-Coutiño , Julia P.G. Jones , Srinivasan Keshav ,...

WSL2 / Windows 11

2023-03-13T00:00:00Z

I naively thought I could just use WSL2 on Windows on my new laptop. But it turned out this was the year of Linux on the Laptop for me. For posterity here’s the crib sheet though. wsl -- set-default-version 2 sudo apt update && sudo apt upgrade - yy sudo apt install locales sudo locale-gen en_GB.UTF-8 sudo apt install - yy emacs-gtk direnv gedit git clone ./..rc-files ./scripts/install.sh Some Windows native packages using Chocolatey choco install signal skype wire slack zoom choco install git choco install rustup.install rust-analyzer python choco install docker choco install powertyos dropbox googledrive wiztree However, choco install texlive didn’t work so well, so I fell back to WSL2: sudo apt install latexmk texlive-latex-base texlive-xetex ttf-mscorefonts-installer . Use MS...

Leveraging Scientific Innovation and AI to Scale Carbon Markets

2023-03-07T00:00:00Z

Discussion with Mantle Labs about carbon credits

A Case for Planetary Computing

2023-03-01T00:00:00Z

Preprint of planetary computing paper Patrick Ferris , Michael Dales , Sadiq Jaffer , Amelia Holcomb , Eleanor Toye Scott , Thomas Swinfield , Alison Eyres , Andrew Balmford , David A Coomes , Srinivasan Keshav and Anil Madhavapeddy . Working paper at arXiv . URL (arxiv.org) DOI BIB

OCaml operators

2023-02-26T00:00:00Z

An OCaml operator cheatsheet for OCaml operators that I have found useful.

Software folklore

2023-02-26T00:00:00Z

Truly cursed bugs that have become software folklore .

Species distribution modelling using CNNs

2023-02-01T00:00:00Z

Species distribution modelling using CNNs This is an idea proposed in 2023 as a Cambridge Computer Science Part III or MPhil project, and has been completed by Emily Morris . It was co-supervised with David A Coomes . The goal of this project is to compare the performance of MaxEnt techniques to the performance of a CNN model for the task of species distribution modeling. The CNN model will use remote sensing data as part of the input features. The remote sensing data we plan on using is a combination of LULC data (e.g. Dynamic World) and satellite imagery (Planet/Landsat 8/Sentinel 2). We will also use more classical environmental variables from WorldClim and soil data. To evaluate it, we will focus on proteas for the species distribution modeling task. We have two observation data...

How to update a Tezos Smart-Contract

2023-01-25T11:28:59Z

How to update a Smart-Contract The Cambridge Center for Carbon Credits (4C) has been looking at using a distributed ledger, aka blockchain, to make a public, immutable, verifiable store of information about carbon-offsets as a way of providing traceability into the validity of that offset. Whilst blockchains are not my domain of expertise, I was tasked earlier this year with getting the project's smart contract ready for production, being the experienced generalist software-engineer on the team. I've previously written up some of the trials of getting started with the Tezos blockchain platform from a software-engineering perspective, and this post is more of the same, but focussed on one particular area: how do you ship updates to your smart-contract once you've shipped it? The problem is this: any bit of software evolves...

Bibliography Management

2023-01-17T00:00:00Z

Published Tue 17 Jan 2023. Last update Wed 7 Jun 2023. I recently received a request to outline my process for managing bibliographies. Here’s a breakdown of my approach. Bibliography Generation I use the Zotero application as a reference manager. The Zotero Firefox connector 1 lets me add papers from URLs. Zotero is quite smart about grabbing the metadata from the URL or PDF itself. The view I have from Zotero is: While Zotero is great for storing local copies of papers and generating bibliographies, I don’t find it to be optimal for organisation and note-taking. Instead, I use the Zotero Better Bibtex plugin to export all the items in my library 2 into a BibLaTeX file that I reference items in with another application. Note-taking I’m using Obsidian as a...

OCaml.org: recapping 2022 and queries on the Fediverse

2023-01-02T00:00:00Z

I recap the OCaml community progress in 2022, which covers a number of bases ranging from the release of OCaml 5.0, the launch of a new website with integrated documentation for 20000+ packages, prototyping new developer workflows that are better integrated into editors, and the launch of ActivityPub based services such as https://watch.ocaml.org .

The role of urban vegetation in human health

2023-01-01T00:00:00Z

The role of urban vegetation in human health This is an idea proposed in 2023 as a Cambridge Computer Science PhD topic, and is currently being worked on by Andres Zuñiga-Gonzalez . It is co-supervised with Ronita Bardhan . Cities around the globe have experienced unprecedented growth in recent years, becoming centres of economic, cultural, and social hubs for human activity. Rapid urbanisation has transformed the physical landscape and significantly altered local climates, with predictions stating that cities will harbour more than 70% of the population by the middle of the 21st century. This has also changed the climatic variables to which humans are most directly exposed. Combining global climatic changes with localised human activities has created new challenges that cities must solve to be...

Reverse emulating agent-based models for policy simulation

2023-01-01T00:00:00Z

Reverse emulating agent-based models for policy simulation This is an idea proposed in 2023 as a Cambridge Computer Science Part III or MPhil project, and has been completed by Pedro Sousa . It was co-supervised with Sadiq Jaffer . Governments increasingly rely on simulation tools to inform policy design. Agent-based models (ABMs) simulate complex systems to study the emergent phenomena of individual behaviours and interactions in agent populations. However, these ABMs force an iterative, time-consuming, unmethodical parameter tuning of key policy "levers" (or input parameters) to steer the model towards the envisioned outcomes. To unlock a more natural workflow, this project investigates reverse emulation , a novel approach that streamlines policy design using probabilistic machine learning...

Remote Sensing of Nature

2023-01-01T00:00:00Z

Remote Sensing of Nature Measuring the world's forest carbon and biodiversity is made possible by remote sensing instruments, ranging from satellites in space (Landsat, Sentinel, GEDI) to ground-based sensors (ecoacoustics, camera traps, moisture sensors) that take regular samples and are processed into time-series metrics and actionable insights for conservation and human development. However, the algorithms for processing this data are challenging as the data is highly multimodal (multispectral, hyperspectral, synthetic aperture radar, or lidar), often sparsely sampled spatially, and not in a continuous time series. I work on various algorithms and software and hardware systems we are developing to improve the datasets we have about the surface of the earth. Mapping nature on earth...

Mapping LIFE on Earth

2023-01-01T00:00:00Z

Mapping LIFE on Earth Human-driven habitat loss is recognised as the greatest cause of biodiversity loss, but we lack robust, spatially explicit metrics quantifying the impacts of anthropogenic changes in habitat extent on species' extinctions. LIFE is our new metric that uses a persistence score approach that combines ecologies and land-cover data whilst considering the cumulative non-linear impact of past habitat loss on species' probability of extinction. We apply large-scale computing to map ~30k species of terrestrial vertebrates and provide quantitative estimates of the marginal changes in the expected number of extinctions caused by converting remaining natural vegetation to agriculture, and also by restoring farmland to natural habitat. We are also investigating many of the...

Interspatial Networking with DNS

2023-01-01T00:00:00Z

Interspatial Networking with DNS This is an idea proposed in 2023 as a Cambridge Computer Science PhD topic, and is currently being worked on by Ryan Gibb . It is co-supervised with Jon Crowcroft . The existing Internet architecture lacks support for naming locations and resolving them to the myriad addressing mechanisms we use beyond IP. While there have been many advances in addressing locations via multiple routing schemes , it remains difficult to refer to location-based services via logical names . This in turn makes it difficult to deploy network services that can be referred to by a stable name that specifies a given location, and that resolves to the addresses of the devices in that space. This matters because there are a broad class of network-connected devices with a physical...

Improving Resilience of ActivityPub Services

2023-01-01T00:00:00Z

Improving Resilience of ActivityPub Services This is an idea proposed in 2023 as a Cambridge Computer Science Part II project, and has been completed by Gediminas Lelešius . The original goal of the project was to improve the resilience of the distributed social networking protocol "ActivityPub", by caching the content on multiple instances and serving them in case the origin instance goes down. The project uses public-key cryptography to ensure data integrity, build a network of public key servers and verifiers and use that consensus instead of relying on individual servers to provide trustworthy data. The core deliverable is a key server gathering and serving public keys, a verifier checking the entries of that server, and a modified Mastodon server rescuing failed ActivityPub requests using an...

Assessing high-performance lightweight compression formats for geospatial computation

2023-01-01T00:00:00Z

Assessing high-performance lightweight compression formats for geospatial computation This is an idea proposed in 2023 as a Cambridge Computer Science Part III or MPhil project, and has been completed by Omar Tanner . It was co-supervised with Sadiq Jaffer . Geospatial data processing can benefit from by applying lightweight compression techniques to data in GeoTIFF format, addressing the challenge of modern CPU bandwidth surpassing RAM bandwidths. This project will explore how to mitigate the impact of poor cache locality and the resulting memory bottlenecks by leveraging CPU superscalar capabilities and SIMD instructions. By implementing SIMD-optimised compression, data can remain compressed in RAM and closer to the CPU caches, facilitating faster access and alleviating memory constraints....

Hacking on the Remarkable 2

2022-12-17T00:00:00Z

Published Sat 17 Dec 2022. Last update Wed 7 Jun 2023. I’ve recently had the good fortune to come into possession of a reMarkable 2 E-Ink writing tablet. This device runs a modified version of Linux, and contains the following message in the copyrights and licenses information: GPLv3 Compliance The General Public License version 3 and the Lesser General Public License version 3 also requires you as an end-user to be able to access your device to be able to modify the copyrighted software licensed under these licenses running on it. To do so, this device acts as an USB ethernet device, and you can connect using the SSH protocol using the username ‘root’ and the password ‘’. As a result of this, there is a vibrant community of hacking for the remarkable. This blog post will walk...

Nix

2022-12-14T00:00:00Z

Published Wed 14 Dec 2022. Last update Mon 3 Feb 2025. Tags: technology . Nix snowflake 1 Nix is a deployment system that uses cryptographic hashes to compute unique paths for components 2 that are stored in a read-only directory: the Nix store, at /nix/store/- . This provides several benefits, including concurrent installation of multiple versions of a package, atomic upgrades, and multiple user environments [1] . Nix uses a declarative domain-specific language (DSL), also called Nix, to build and configure software. The snippet used to deploy the DNS server is in fact a Nix expression. This example doesn’t demonstrate it, but Nix is Turing complete. Nix does not, however, have a type system. We use the DSL to write derivations for software, which...

Hillingar: MirageOS Unikernels on NixOS

2022-12-14T00:00:00Z

Published Wed 14 Dec 2022. Last update Mon 3 Feb 2025. Tags: projects . A version of this blog post can be found on the Tarides website: https://tarides.com/blog/2022-12-14-hillingar-mirageos-unikernels-on-nixos . An arctic mirage [1] 1 Introduction The Domain Name System (DNS) is a critical component of the modern Internet, allowing domain names to be mapped to IP addresses, mailservers, and more 2 . This allows users to access services independent of their location in the Internet using human-readable names. We can host a DNS server ourselves to have authoritative control over our domain, protect the privacy of those using our server, increase reliability by not relying on a third party DNS provider, and allow greater customization of the records served (or the behaviour of the server itself). However, it can be quite...

Financing Forests: A Credible Approach towards Halting Tropical Deforestation

2022-11-16T00:00:00Z

Wednesday seminar on financing forests using carbon credits

17th William Pitt Seminar - Who's in Charge?

2022-11-01T00:00:00Z

I opened the 17th William Pitt Seminar at Pembroke College on climate change with a brief talk about the status of the world's biodiversity, and how we have more agency than ever before to take matters into our own hands.

Verifying distributed systems with Isabelle/HOL

2022-10-12T01:00:00Z

This post also appears on Larry Paulson’s blog . We use distributed systems every day in the form of internet services. These systems are very useful, but also challenging to implement because networks are unpredictable. Whenever you send a message over the network, it is likely to arrive quite quickly, but it’s possible that it might be delayed for a long time, or never arrive, or arrive several times. When you send a request to another process and don’t receive a response, you have no idea what happened: was the request lost, or has the other process crashed, or was the response lost? Or maybe nothing was lost at all, but a message has simply been delayed and may yet arrive. There is no way of knowing what happened, because unreliable message-passing is the only way how processes can communicate....

The 2nd ed of Real World OCaml is available in shops

2022-10-01T01:00:00Z

I'm delighted to report that the second edition of Real World OCaml is now available from Cambridge University Press! It's also freely available online , and CUP also kindly agreed that the PDF version could be freely available online thanks to sponsorship from Tarides . [… 105 words ] Anil Madhavapeddy and Yaron Minsky . Book published by Cambridge University Press . URL (cambridge.org) DOI BIB

Homogeneous Builds with OBuilder and OCaml

2022-09-01T01:00:00Z

Paper on our incremental computation DSL ocurrent presented in OCaml Workshop 2022 Tim McGilchrist , David Allsopp , Patrick Ferris , Antonin Décimo , Thomas Leonard , Anil Madhavapeddy and Kate Deplaix . Paper in the proceedings of the 2022 OCaml Users and Developers Workshop . URL (icfp22.sigplan.org) BIB

Nim

2022-08-30T01:00:00Z

Published Tue 30 Aug 2022. Last update Tue 30 Aug 2022. Tags: projects . I recently had the good fortune to attended a talk by Simon Peyton Jones titled “Immutability changes everything: 40 years of functional programming” which chronologic Simon’s life and research career. Simon is a self-described one trick pony, where that trick is functional programming. However, this trick has taken him from being a major instigator in Haskell - he remains a lead developer of the Glasgow Haskell Compiler to this day - to Microsoft research, and more recently to Epic Games working on a new programming language dubbed ‘ Verse ’. In this talk Simon described how he first got interested in Computer Science through a problem posed to him: the game Nim. 1 The game Picture the scene: you and...

A DSL for decentralised identity in OCaml

2022-08-01T01:00:00Z

A DSL for decentralised identity in OCaml This is an idea proposed in 2022 as a Cambridge Computer Science Part II project, and has been completed by Michał Mgeładze-Arciuch . It was co-supervised with Patrick Ferris . There are currently multiple identity providers without direct incentives to cooperate. This leads to many redundant implementations of the identity handling logic, many of which are not immediately compatible with each other, leading to additional increases in friction when eventual agreement needs to be reached to perform user actions. Furthermore, from the perspective of the user of the identity service, they need to keep track of identity documents from multiple sources, which leads to more security attack surface. Solving the problem of partial identity proofs allows for many possible...

Trusted Carbon Credits

2022-05-06T01:00:00Z

With the recent controversies over low-integrity carbon credits, I spoke to Vox magazine about my skepticism about Adam Neumann's new startup. "The problem with the current markets is nothing to do with how we can trade these more effectively," said Anil Madhavapeddy, who is an associate professor of computer science and technology at Cambridge University and the director of the Cambridge Center for Carbon Credits. "We just do not have enough supply." -- Vox The Cambridge Centre for Carbon Credits is an initiative I started with Andrew Balmford , David A Coomes , Srinivasan Keshav and Thomas Swinfield , aimed at issuing trusted and verifiable carbon credits towards the prevention of nature destruction due to anthropogenic actions. We researched a combination of large-scale data processing (satellite and...

Secure Programming with Dispersed Compartments

2022-05-01T01:00:00Z

Secure Programming with Dispersed Compartments This is an idea proposed in 2022 as a Cambridge Computer Science PhD topic, and has been completed by Zahra Tarkhani . This PhD project proposes novel approaches and mechanisms for application compartmentalization and isolation to reduce their ever-growing attack surfaces. Our approach is motivated by the key observation that while hardware vendors compete to provide security features (notably memory safety and privilege separation) existing systems software like commodity OSs fail to utilize such features to improve application security and privacy properly. We propose a novel principled approach to privilege separation and isolation, enabling application security to be designed and enforced within and across different isolation boundaries, and...

OCaml Multicore Monthly: heading towards OCaml 5.0

2022-04-19T01:00:00Z

We're getting closer to a stable release of OCaml 5.0, including reenabling support for the BSDs and introducing ARM64 multicore support.

Virtual and Augmented Reality

2022-04-18T01:00:00Z

Published Mon 18 Apr 2022. Last update Tue 19 Apr 2022. I’ve been interested in virtual reality (VR) and augmented reality (AR) recently as the use case for the system(s) I’m working on in my MPhil project. This blog post will cover my development experience in porting my cubes game to both VR and AR, and give a small demo for the latter. Microsoft Windows was used for ease of development with Unity and as it’s a hard requirement for the AR headset made by Microsoft. VR The Oculus Quest 2 from Meta (formally Facebook) is a VR headset running an Android derivative. This means you can do cool stuff like run Termux on it (an Android terminal emulator and unvirtualised Linux environment). It’s nowhere near as flexible as something like what SimulaVR are doing, which features a...

OCaml Multicore Monthly: post merge activites

2022-02-09T00:00:00Z

After we got the massive OCaml 5.0 pull request merged, we've taken some time to consolidate the trunk branch of OCaml and start down the release path towards getting OCaml 5.0 out of the door.

DNS LOC

2022-01-18T00:00:00Z

Published Tue 18 Jan 2022. Last update Sat 21 May 2022. The Domain Name System (DNS) has a little known ‘LOC’ resouce record (RR) defined in 1996 for encoding location information purportedly for hosts, networks, and subnets[0]. It encodes: Latitude Longitude Altitude Size of the referenced sphere Horizontal and vertical precision A number of suggested uses are proposed in the RFC: USENET - a distributed discussion system now ecplised by the World Wide Web - geographic flow maps. A ‘visual traceroute’ application showing geographical flow of IP packet, presumably relying on router responding to both IP TTL timeouts and DNS LOC requests. Network management based using LOC RRs to map hosts and routers. RFC-1876 also describes algorithms for resolving locations for domain names or IP...

Book Review: The Future of Fusion Energy

2022-01-03T00:00:00Z

I give a five-star ⭐️⭐️⭐️⭐️⭐️ rating to the following book: Jason Parisi and Justin Ball. The Future of Fusion Energy . World Scientific, 2019. ISBN 978-1-78634-749-7. Available from Amazon US , Amazon UK , and many other retailers. I came to this book looking for answers to questions such as: Is there still hope that a fusion power plant will ever be viable? If so, what exactly are the main obstacles on the way there? Why has progress in this field been so slow? And what should I make of the various startups claiming to have a fusion power plant just round the corner? The book provides an excellent, detailed answer to these questions, and more. It’s the best kind of popular science book: you don’t need a physics degree to read it, but it doesn’t fob you off with oversimplified...

Using effect handlers for efficient parallel scheduling

2022-01-01T00:00:00Z

Using effect handlers for efficient parallel scheduling This is an idea proposed in 2022 as a Cambridge Computer Science Part III or MPhil project, and has been completed by Bartosz Modelski . Modern hardware is so parallel and workloads are so concurrent, that there is no single, perfect scheduling strategy across a complex application software stack. Therefore, there are significant performance advantages to be gained from customizing and composing schedulers. Multicore parallelism is here to stay, and in contrast with clock frequency increases, schedulers have to be carefully crafted in order to take full advantage of horizontal scaling of the underlying architecture. That’s because designs need to evolve as synchronization primitives such as locks or atomics do not scale endlessly to many...

Spatial Name System

2022-01-01T00:00:00Z

Spatial Name System This is an idea proposed in 2022 as a Cambridge Computer Science Part III or MPhil project, and has been completed by Ryan Gibb . It was co-supervised with Jon Crowcroft . The development of emerging classes of hardware such as Internet of Thing devices and Augmented Reality headsets has outpaced the development of Internet infrastructure. We identify problems with latency, security and privacy in the global hierarchical distributed Domain Name System. To remedy this, we propose the Spatial Name System, an alternative network architecture that relies on the innate physicality of this paradigm. Utilizing a device’s pre-existing unique identifier, its location, allows us to identify devices locally based on their physical presence. A naming system tailored to the physical...

Scalable agent-based models for optimized policy design

2022-01-01T00:00:00Z

Scalable agent-based models for optimized policy design This is an idea proposed in 2022 as a Cambridge Computer Science Part III or MPhil project, and has been completed by Sharan Agrawal . It was co-supervised with Srinivasan Keshav . As the world faces twinned crises of climate change and biodiversity loss, the need for integrated policy approaches addressing both is paramount. To help address this, this project investigates a new agent-based model dubbed the VDSK-B. Using Dasgupta's review of the economics of biodiversity , it builds on the Dystopian Schumpeter meets Keynes (DSK) climate economics model to link together the climate, economy and biosphere. This is the first ABM proposed that integrates all 3 key elements. The project also investigates how to scale such ABMs to be applicable for global policy design and scale to planetary-sized...

Planetary Computing

2022-01-01T00:00:00Z

Planetary Computing Planetary computing is our research into the systems required to handle the ingestion, transformation, analysis and publication of global data products for furthering environmental science and enabling better informed policy-making. We apply computer science to problem domains such as forest carbon and biodiversity preservation, and design solutions that can scalably processing geospatial data that build trust in the results via traceability and reproducibility. Key problems include how to handle continuously changing datasets that are often collected across decades and require careful access and version control. "Planetary computing" originated as a term back in 2020 when a merry band of us from Computer Science ( Srinivasan Keshav and me, later joined by Sadiq Jaffer ,...

Enhancing the Security & Privacy of Wearable Brain-Computer Interfaces

2022-01-01T00:00:00Z

Preprint on security vulnerabilities in brain-computer interfaces Zahra Tarkhani , Lorena Qendro , Malachy O'Connor Brown , Oscar Hill , Cecilia Mascolo and Anil Madhavapeddy . Working paper at arXiv . URL (arxiv.org) DOI BIB PDF

Decentralised Capability-based Code Collaboration using Matrix

2022-01-01T00:00:00Z

Decentralised Capability-based Code Collaboration using Matrix This is an idea proposed in 2022 as a Cambridge Computer Science Part II project, and has been completed by Samuel Wedgwood . In 2005, due to licensing disputes, the team behind Linux parted ways with their proprietary source management tool BitKeeper, and needed a new solution. This prompted the development of Git, an open-source decentralised version control system (DVCS), which was soon used to manage the source code of Linux. Contributions were submitted as patch files, which contained just the differences that the contribution made, to an email list, which were reviewed and applied to the central Git repository for Linux. Git grew in popularity and other projects started using it to manage their source code. Then, in 2008, the...

OCaml Multicore Monthly: code review complete with Inria

2021-12-21T00:00:00Z

We've been working hard on OCaml multicore support, and went over to Paris to sit down with some core developers from Inria and work through code review of our proposed patches.

2021 SDG Hackathon

2021-11-06T00:00:00Z

I participated in the 2021 SDG Hackathon hosted by CORELAID Switzerland and the University of Basel. The data used was obtained from text analysis of projects realted to SDGs in Switzerland using five different systems. Each system detected a number of times each SDG in each project. I wanted to show how commonly each SDG is detected by each system from the dataset and for this purpose I created a rank view of the 17 SDGs using ggbump and integrating it with ggtext and several other packages. You can check out the code in the GitHub repo .

Launching the Cambridge Centre for Carbon Credits

2021-11-04T00:00:00Z

I launched 4C recently, and Pembroke College covers the launch with an interview with me. The world is facing a large-scale environmental crisis. Two parallel and related strands of this are, first, the crisis in biodiversity and the rapid extinction of many species, recently addressed at the COP15 UN Biodiversity Conference in October, and second, the threat of climate change, the topic of last month’s COP26 summit in Glasgow. Pressure is growing on governments to execute nature-based solutions which will offset some of the most damaging impacts of these crises. While COP26 built some momentum, there is still a long way to go to turn promises into lasting change. More engagement with the private sector is urgently needed. The solution to the crisis is two-pronged: we must engage...

What is an Operating System?

2021-11-03T00:00:00Z

I am the latest person to feature on the first season of the Signals and Threads podcast hosted by Yaron Minsky (you may recognise him as my co-author on Real World OCaml ). Anil Madhavapeddy is an academic, author, engineer, entrepreneur, and OCaml aficionado. In this episode, Anil and Ron consider the evolving role of operating systems, security on the internet, and the pending arrival (at last!) of OCaml 5.0. They also discuss using Raspberry Pis to fight climate change; the programming inspiration found in British pubs and on Moroccan beaches; and the time Anil went to a party, got drunk, and woke up with a job working on the Mars Polar Lander. -- Signals and Threads I think I might be the first non- Jane Street person to be on their podcast! Quite the honour.

Hosting HotCRP

2021-10-17T01:00:00Z

I once had cause to setup HotCRP for local hosting. Specifically on a local Lab-hosted VM image. Some of what follows is specific to the CUCL VM hosting service, but I think most of it is HotCRP generic and so may be of use. Anyway, here’s the crib sheet, starting from https://mbtech.github.io/Setting-up-hotcrp/ … # setup some variables YOUR-DOMAIN = " hotcrp-test.cl.cam.ac.uk " YOUR-WORKSHOP = " sysws18 " YOUR-PASSWORD = " mybestpassword " YOUR-EMAIL = " postmaster@example.com " Lab specifics Assume we start from a default Ubuntu template VM, and then… Configure the VM cl-asuser passwd # set UNIX password for sudo # create some space for d in /usr/src/ * ; do export K = $ ( uname - r | sed ' s/-generic$// ' ) echo - n $ K $ d ... case $ d in " /usr/src/linux-headers- $ K " | " /usr/src/linux-headers- $ { K } -generic "...

ILNP Interface

2021-10-16T01:00:00Z

Published Sat 16 Oct 2021. Last update Sat 22 Oct 2022. Tags: projects . My dissertation involved implementing an Identifier-Locator Network Protocol (ILNP) overlay network in Python which can be found at github.com/RyanGibb/ilnp-overlay-network . As part of this, I wanted to add an application layer interface to the overlay to support existing applications. (To those who still want to know why I posit, why not?) That is, applications other than those written in python specifically for the overlay. This would also allow multiple applications to run over one overlay network stack. However, this wasn’t a priority for my dissertation as it wasn’t necessary to obtain experimental results. Since graduating I’ve found a few weekends to work on this and a solution will be explored in this blog post. Datagrams First up, how can we send a datagram over...

OCaml Multicore Monthly: effect handling confirmed for 5.0

2021-10-01T01:00:00Z

We're making steady progress on getting multicore support merged into OCaml, including some great developer meetings where we achieved consensus with the core team to include support for effect handlers in the 5.0 release.

Forest preservation and restoration

2021-09-25T01:00:00Z

I track external notes and media articles here on forest preservation and restoration as part of my work on Trusted Carbon Credits . Not complete, just a reading list. Can YOU Fix Climate Change? (great short summary of the overall issues) Rewilding Affric Highlands initiative to rewild Scotland over 30 years Gabon's Climate Law Pairing agroforestry with livestock: the major benefits Major global companies to fund nature restoration projects in UK's national parks (via Palladium group ) Remote sensing Remote sensing of plant biodiversity Geobon - global researcher network working on above. Earth-i - sub-1m sensing satellite constellation Mantle Labs - earth observation + machine learning for farmers Seagrass from space Keshav's comms technologies for energy informatics Carbon Credits Notes on Carbon offsets for scientific societies Voluntary Carbon Markets integrity initiative Voluntary Carbon Markets Rocket in 2021, On Track to Break $1B for First Time Biodiversity Biodiversity: Perspectives of a Techie - Dave Thau - Data and Technology Global Lead Scientist, WWF Valuing climate change Cryptocarbon: The promises and pitfalls of forest protection on a blockchain Valuing climate damages at the country level - nature climate change, 2018 Country-level social cost of carbon , nature climate change...

Decentralised tech on Recoil

2021-09-19T01:00:00Z

Nick Ludlam and I have self-hosted recoil.org since around 1996, typically for email and web. These days, there are a number of interesting software stacks around decentralised communication that we deploy. This note keeps track of them. Email (active) Currently Postfix and DKIM/SPIF relays Till 2019, was OpenSMTPD and would like to return to it but waiting on filter support. Till around 2016, was qmail but finally gave up due to difficulty of spam filtering. Next step will be to try out the MirageOS email stack that dinosaure has been leading the development of. Web (active) This website is an OCaml webserver running a custom multicore OCaml webserver Next step will be to go solar powered with a custom DNS server. DNS (inactive) MirageOS DNS server. Currently offline due to a hosting...

Several podcast interviews

2021-09-01T01:00:00Z

I regularly get asked to give interviews on the topics that I work on, especially for podcasts. To make them easier to find for anybody who’s interested, I thought I would make a list. They touch on a range of different topics, although there is also some overlap so I wouldn’t recommend listening to them all in a row! (By the way, if you want a list of conference talks I have given, I have a YouTube playlist for that.) Here’s a list of interviews I’ve given as of September 2021: Interview with Wix Engineering , in which we discuss my book, the state of Automerge, the convergence of streaming systems and databases, Kafka’s move to replace ZooKeeper with their own Raft implementation, impact of my research, and more. Recorded 16 June 2021, published 26 August 2021. Video , transcript . Interview...

Roadmap for OCaml's online presence

2021-08-27T01:00:00Z

After a decade of good service, it's time to overhaul OCaml's online presence to more modern technologies. This post lays out the roadmap for the third edition of the OCaml.org website.

Tarballing some files

2021-08-25T01:00:00Z

I recently discovered, to some irritation, that the --include PATTERN option to tar seems only to apply to directories – and if the PATTERN doesn’t match, it won’t traverse subdirectories. But I wanted to include *.php for some reason. So instead pipe the output of find , or better these days, fd : fd - e php - 0 | tar - cvjf TARBALL.bz2 -- null -- files-from -

How Computer Science Can Aid Forest Restoration

2021-08-01T01:00:00Z

Preprint about our working notes on how CS might contribute to forest preservation Gemma Gordon , Amelia Holcomb , Tom Kelly , Srinivasan Keshav , Jon Ludlam and Anil Madhavapeddy . Working paper at arXiv . URL (arxiv.org) DOI BIB PDF

Experiences with Effects

2021-08-01T01:00:00Z

Paper on programming with effects in OCaml Thomas Leonard , Craig Ferguson , Patrick Ferris , Sadiq Jaffer , Tom Kelly , KC Sivaramakrishnan and Anil Madhavapeddy . Paper in the proceedings of the 2021 OCaml Users and Developers Workshop . URL (icfp21.sigplan.org) BIB PDF

Cubes

2021-06-03T01:00:00Z

Published Thu 3 Jun 2021. Last update Thu 3 Mar 2022. Tags: projects . I was teaching myself Unity the week after the last exam for my undergrad and ended up creating this. It’s an infinite runner with cubes created in unity. Play it here . The repo can be found at github.com/RyanGibb/cubes . Re-rooting To allow the game to be infinite, whilst also avoiding floating point errors, the game world is periodically ‘re-rooted’. This involves moving the player and all objects back a certain distance in one frame. Objects behind the player can be safely removed. However, it may prove challenging to get to a distance where floating point errors occur! World Generation The world is generated using a Markov Chain containing transitions from segments to segments. Segments may be prefabricated or...

Retrofitting effect handlers onto OCaml

2021-06-01T01:00:00Z

Paper on retrofitting effects in OCaml presented in PLDI 2024 KC Sivaramakrishnan , Stephen Dolan , Leo White , Tom Kelly , Sadiq Jaffer and Anil Madhavapeddy . Paper in the proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation . URL (dl.acm.org) DOI BIB PDF

Network layer mobility

2021-05-10T01:00:00Z

Published Mon 10 May 2021. Last update Mon 24 Mar 2025. Tags: projects , research . My undergraduate dissertation, “Ubiquitous Communication for the Internet of Things: An Identifier-Locator addressing split overlay network”, explored how we can modify the Internet Protocol to better support resource-constrained highly mobile embedded devices. A copy can be found here (or here for St Andrews folk), and the associated implementation can be found at ilnp-overlay-network . Network stack First, some prerequisite networking knowledge. A network protocol stack is a view of how protocols are organised into layers. The OSI model describes named layers; including the physical, link, network, transport, and application layers. Note the OSI model and TCP/IP have differing semantics , but this is...

25 years of OCaml!

2021-05-09T01:00:00Z

On this day in 1996, Xavier Leroy announced Objective Caml 1.00 (the language wasn’t officially called OCaml until 4.00.0 in July 2012 ). I wouldn’t start using OCaml for another 7 years; I think I may have dropped Research Machines Basic by then and was mucking around with a mix of Visual Basic, Turbo Pascal and Delphi, but I hadn’t yet got an email address either. For whatever reason, the 1.00 tag deletes the boot images, but they can be taken from the commit before . I could get the runtime to build quite easily on Ubuntu, but alas while it appeared to be able to run boot/ocamllex , boot/ocamlc was just segfaulting. I debated firing up my old dual-Pentium III which apparently had Slackware 10 on it, but I thought I’d give the Windows port a go first! According to the docs, the Windows compiler used to be...

It's time to say goodbye to the GPL

2021-04-14T01:00:00Z

The trigger for this post is the reinstating of Richard Stallman, a very problematic character , to the board of the Free Software Foundation (FSF). I am appalled by this move, and join others in the call for his removal. This occasion has caused me to reevaluate the position of the FSF in computing. It is the steward of the GNU project (a part of Linux distributions, loosely speaking ), and of a family of software licenses centred around the GNU General Public License (GPL). These efforts are unfortunately tainted by Stallman’s behaviour. However, this is not what I actually want to talk about today. In this post I argue that we should move away from the GPL and related licenses (LGPL, AGPL), for reasons that have nothing to do with Stallman, but simply because I think they have failed to achieve their purpose, and they are more...

Building the future of computing, with your help

2021-02-23T00:00:00Z

For the last five or six years, since I bid goodbye to the startup scene and Silicon Valley, I have been increasingly working in public. I have written a book , given around 100 talks (many of which are available on YouTube ), published over 20 research papers (all freely available from my website), and released and maintained some open source projects . Just a few months ago I released a new undergraduate-level course on distributed systems, consisting of 7 hours of video lectures and 87 pages of notes and exercises, all free; in student evaluation at the University of Cambridge , over 80% rated my lectures and notes as “excellent”. I love doing first-rate work and making it broadly available. In fact, apart from my book, I give everything away for free, because I want to be able to reach and help the broadest possible set of people. And even my book is very cheap...

Decentralised content moderation

2021-01-13T00:00:00Z

Who is doing interesting work on decentralised content moderation? With Donald Trump suspended from Twitter and Facebook, and Parler kicked off AWS, there is renewed discussion about what sort of speech is acceptable online, and how it should be enforced. Let me say up front that I believe that these bans were justified. However, they do raise questions that need to be discussed, especially within the technology community. As many have already pointed out, Twitter, Facebook and Amazon are corporations that are free to enforce their terms of service in whatever way they see fit, within the bounds of applicable law (e.g. anti-discrimination legislation). However, we should also realise that almost all social media, the public spaces of the digital realm, are in fact privately owned...

Void Processes: Minimising privilege by default

2021-01-01T00:00:00Z

Void Processes: Minimising privilege by default This is an idea proposed in 2021 as a Cambridge Computer Science Part III or MPhil project, and has been completed by Jake Hillion . Void processes intend to make it easier for all developers to produce effectively privilege separated applications. The project has two primary goals: show the merits of starting from zero privilege, and provide the utilities to make this feasible for the average developer. Building void processes involves first reliably removing all privilege from a process then systematically adding back in what is required, and no more. This project utilises Linux namespaces to revoke privilege from an application, showing how this can be done and why its easier in some domains than others. It then shows how to inject...

Security analysis of brain-computing interfaces

2021-01-01T00:00:00Z

Security analysis of brain-computing interfaces This is an idea proposed in 2021 as a good starter project, and has been completed by Malachy O'Connor Brown and Oscar Hill . It was co-supervised with Zahra Tarkhani and Lorena Qendro . Brain Computing Interface (BCI) technologies, both invasive and non-invasive, are increasingly used in a wide range of applications, from health-care to smart communication and control. Most BCI applications are safety-critical or privacy-sensitive. However, the infinite potentials of BCI and its ever-growing market size have been distracted the BCI community from significant security and privacy threats. In this research, we first investigate the security and privacy threats of various BCI devices and applications, from machine learning adversarial threats to untrusted...

Exploring Concurrency in Agent-Based Modelling with Multicore OCaml

2021-01-01T00:00:00Z

Exploring Concurrency in Agent-Based Modelling with Multicore OCaml This is an idea proposed in 2021 as a Cambridge Computer Science Part II project, and has been completed by Martynas Sinkievič . Computational modelling techniques such as ABMs are used to understand the dynamics of ecosystems and predict their behaviour in response to climate change and ecological disturbances, while also searching for optimal paths towards solutions to these problems. Terrestrial biosphere models are one such model which simulate the vegetation and soil life cycle. There have been two approaches taken with such modelling: The top-down approach take coarse-grained dynamic models that simulate environments in large chunks and scale to large areas as needed, but with a lack of accuracy in the simulated environment...

Using Bloom filters to efficiently synchronise hash graphs

2020-12-02T00:00:00Z

This blog post uses MathJax to render mathematics. You need JavaScript enabled for MathJax to work. In some recent research, Heidi and I needed to solve the following problem. Say you want to sync a hash graph, such as a Git repository, between two nodes. In Git, each commit is identified by its hash, and a commit may include the hashes of predecessor commits (a commit may include more than one hash if it’s a merge commit). We want to figure out the minimal set of commits that the two nodes need to send to each other in order to make their graphs the same. You might wonder: isn’t this a solved problem? Git has to do this every time you do git pull or git push ! You’re right, and some cases are easy, but other cases are a bit trickier. What’s more, the algorithm used by Git is not...

New courses on distributed systems and elliptic curve cryptography

2020-11-18T00:00:00Z

I have just published new educational materials that might be of interest to computing people: a new 8-lecture course on distributed systems, and a tutorial on elliptic curve cryptography. Distributed Systems Since last year I have been delivering an 8-lecture undergraduate course on distributed systems at the University of Cambridge. The first time I delivered it, I inherited the slides and exercises from the people who lectured it in previous years (Richard Mortier, Anil Madhavapeddy, Robert Watson, Jean Bacon, and Steven Hand), and I just used those materials with minor modifications. It was a good course, but it was getting quite dated (e.g. lots of material on CORBA , which is now of mostly historical interest). Therefore, this year I decided to do a thorough refresh of the...

Banyan: Coordination-Free Distributed Transactions over Mergeable Types

2020-11-01T00:00:00Z

Paper on Banyan for coordination-free distributed transactions in ASPLAS 2020 Shashank Shekhar Dubey , KC Sivaramakrishnan , Thomas Gazagnaire and Anil Madhavapeddy . Paper in the proceedings of the Asian Symposium on Programming Languages and Systems . URL (link.springer.com) DOI BIB PDF

Using memtrace on Windows

2020-10-08T01:00:00Z

It is said that good things come to those who wait. Jacques-Henri Jourdan demonstrated Statistically profiling memory in OCaml at the 2016 OCaml Workshop and experimental branches of it existed for OCaml 4.03–4.07 in opam. Parts of the work were merged in OCaml 4.10 and the final sections landed with 4.11. The compiler provides support in the form of a series of hooks in various parts of the runtime’s allocation and garbage collection routines. A few weeks ago, Jane Street released memtrace and Memtrace viewer and earlier this week blogged about its use . I thought I’d quickly share the experience of using it on native Windows. At present, although the memtrace library is portable, the memtrace-viewer itself is not (as it happens, this is simply down to Core’s command line parser pulling in a little too much Unix-specific stuff,...

Data visualisation links

2020-09-07T01:00:00Z

Some possibly useful data visualisation links: https://medium.economist.com/why-you-sometimes-need-to-break-the-rules-in-data-viz-4d8ece284919 https://colororacle.org/

The OCaml Platform: 2020

2020-09-01T01:00:00Z

Annual update on the OCaml Platform at the OCaml Workshop Anil Madhavapeddy . Paper in the the 10th ACM OCaml Users and Developers Workshop . URL (icfp20.sigplan.org) BIB

State of the OCaml Platform 2020

2020-08-28T01:00:00Z

Talk on the state of the OCaml Platform in 2020

Renovating an old media PC

2020-08-28T01:00:00Z

Some notes from my first attempt to renovate an old media PC that had a SYSLINUX install without any package management, and a crufty BIOS. Probably outdated now, but I may go back to it one day… First, some background links: https://en.wikipedia.org/wiki/Cylinder-head-sector https://en.wikipedia.org/wiki/FAT_boot_sector https://en.wikipedia.org/wiki/Logical_Block_Addressing#CHS_conversion https://en.wikipedia.org/wiki/Master_Boot_Record https://en.wikipedia.org/wiki/Volume_boot_record https://wiki.archlinux.org/index.php/Syslinux https://wiki.syslinux.org/wiki/index.php?title=Common_Problems#Failed_to_load_ldlinux https://wiki.syslinux.org/wiki/index.php?title=Hardware_Compatibility#USB_related_problems https://wiki.syslinux.org/wiki/index.php?title=Hdt_(Hardware_Detection_Tool) I explored two ways forward: SYSLINUX and GRUB . SYSLINUX I found that getting SYSLINUX working required moving the partition to 0/1/1 – using sectors per track of 63 or 32, and heads per cylinder or 16 or 64 with appropriate cylinder values simply didn’t help. Diagnosed by observing that console displayed only CRLF but no banner – SYSLINUX code ends up with the banner to be displayed just falling into the second sector on the disk, so it can’t be read...

OCaml Multicore Monthly: systhreads compatibility merged

2020-08-20T01:00:00Z

The big advance in the multicore OCaml branch is that we restored compatibility with the traditional OCaml systhreads. This in turn means that many existing software packages just work out of the box on the new runtime. Big news this month is that the systhreads compatibility support PR has been merged, which means that Dune (and other users of the Thread module) can compile out of the box. You can now compile the multicore OCaml fork conveniently using the new opam compiler plugin (see announcement). -- me, on the discussion forum

Retrofitting parallelism onto OCaml

2020-08-01T01:00:00Z

Won best paper award at ICFP 2020 for our paper on retrofitting parallelism onto OCaml! KC Sivaramakrishnan , Stephen Dolan , Leo White , Sadiq Jaffer , Tom Kelly , Anmol Sahoo , Sudha Parimala , Atul Dhiman and Anil Madhavapeddy . Journal paper in Proceedings of the ACM on Programming Languages (vol 4 issue ICFP). URL (dl.acm.org) DOI BIB PDF

Parallelising your OCaml Code with Multicore OCaml

2020-08-01T01:00:00Z

Paper on how to parallelise OCaml code at the OCaml Workshop Sadiq Jaffer , Sudha Parimala , KC Sivaramakrishnan , Tom Kelly and Anil Madhavapeddy . Paper in the proceedings of the 2020 OCaml Users and Developers Workshop . URL (github.com) BIB PDF

OCaml-CI: A Zero-Configuration CI

2020-08-01T01:00:00Z

Presented the new OCaml DSL for continuous integration at the OCaml Workshop Thomas Leonard , Craig Ferguson , Kate Deplaix , Magnus Skjegstad and Anil Madhavapeddy . Paper in the proceedings of the 2020 OCaml Users and Developers Workshop . URL (icfp20.sigplan.org) BIB

Making GPS accurate in dense forests using sensor fusion

2020-08-01T01:00:00Z

Making GPS accurate in dense forests using sensor fusion This is an idea proposed in 2020 as a good starter project, and has been completed by Keshav Sivakumar . It was co-supervised with Srinivasan Keshav and David A Coomes . Current GPS solutions are either very expensive ($8k+) or have relatively poor accuracies (10m+) under dense forest canopy. This project explores how to determine our location accurately in a forest area where we travel by foot under canopy without a GPS signal. What low cost solutions exist to perform localisation under such circumstances? What are the rough accuracies of these solutions? What constraints and advantages do these solutions have (in terms of power, light, cost, etc) We observe that a lot of SLAM algorithms exist these days, but most of the recent research is on...

Discord discord

2020-08-01T01:00:00Z

So for some reason I wanted to do this – use Discord on an iPad without installing the app. This proved surprisingly tricky as Safari on the iPad really wanted to make you use the app and certainly wouldn’t display the desktop site. However, Firefox Focus can though it still forced you into the app from the link in the invite email. However, you can go to the website to signup and then manually invite yourself via a real desktop browser. And this then means that Firefox Focus on the iPad believes in the new account and just lets you in. Seems something of a palaver but hey, these twisted webs we weave.

OCaml Multicore Monthly: preprint paper available

2020-04-27T01:00:00Z

In the April OCaml multicore monthly, we have a preprint available of our ICFP submission about the OCaml 5 multicore runtime. (Update: This paper actually won the ICFP best paper award later in the year! Read it at " Retrofitting parallelism onto OCaml ").

The First Billion Real Deployments of Unikernels

2020-02-26T00:00:00Z

Part 2

Rebuilding Operating Systems with Functional Principles

2020-02-26T00:00:00Z

Delivered the distinguished seminar series at St Andrews on rebuilding Operating Systems with functional principles

Programming the Next Trillion Embedded Devices

2020-02-26T00:00:00Z

Part 3

OCaml Multicore Monthly: starting upstream to OCaml

2020-01-29T00:00:00Z

We started the process of upstreaming our multicore OCaml branch to mainline OCaml, and so I started posting regular updates to the community forum. The most common question we get is how to contribute to the overall multicore effort. As I noted last year, we are now in the process of steadily upstreaming our efforts to mainline OCaml. Therefore, the best way by far to contribute is to test for regressions or opportunities for improvements in the patches that are outstanding in the main OCaml repository. -- me, on the discussion forum

Information Flow for Trusted Execution

2020-01-01T00:00:00Z

Information Flow for Trusted Execution There is now increased hardware support for improving the security and performance of privilege separation and compartmentalization techniques such as process-based sandboxes, trusted execution environments, and intra-address space compartments. We dub these "hetero-compartment environments" and observe that existing system stacks still assume single-compartment models (i.e. user space processes), leading to limitations in using, integrating, and monitoring heterogeneous compartments from a security and performance perspective. This project explores how we might deploy techniques such as fine-grained information flow control (DIFC) to allow developers to securely use and combine compartments, define security policies over shared system resources, and...

Setting up a Macbook

2020-01-01T00:00:00Z

This is the set of things that I roughly did to set up my old new Macbook: configurations, packages, etc. Not guaranteed complete, but hopefully captures much of it. Keyboard Set the touchbar to expanded Turn off various emacs conflicts (e.g., S-M-6 ) Use Karabiner Elements to make CAPSLOCK become ESC on a single press Homebrew packages Install Homebrew and then # install packages brew install \ aspcud \ aspell \ bash-completion \ coreutils \ direnv \ emacs \ evernote \ ffmpeg \ font-hack-nerd-font \ gawk \ gcc \ get_iplayer \ ghostscript \ git \ git-lfs \ gnupg2 \ gnuplot \ gpg-agent \ graphviz \ imagemagick \ jq \ lua \ mu \ ncftp \ nmap \ ocaml \ offlineimap \ omnigraffle \ opam \ python \ python@2 \ qemu \ rcs \ readline \ rsync \ socat \ sshfs \ telnet \ tmux \ unrar \ wget # install casks brew...

OpenBSD cloud hosting options

2019-08-29T01:00:00Z

I asked on Twitter about hosting options for OpenBSD on cloud providers, so that we could have some alternative options for Recoil. We have a strong preference for bare-metal and not VMs when it comes to OpenBSD. Options that came back were: OpenBSDAMS Dedicated/VMs for openbsd hosting (see here ) Mythic Beasts I have provisioned a bare metal server there and they kindly stuck a USB stick in with an OpenBSD installer. DataCentreLite Not tried this yet but possible followup . LiquidWeb Good recommendation from Stephane

The OCaml Platform in 2019

2019-08-01T01:00:00Z

Annual update on the OCaml Platform in 2019 Anil Madhavapeddy and Gemma Gordon . Paper in the proceedings of the OCaml Workshop 2019 . URL (icfp19.sigplan.org) BIB

MirageOS 4: the dawn of practical build systems for exotic targets

2019-08-01T01:00:00Z

Paper on the MirageOS 4 build system at OCaml Workshop Lucas Pluvinage , Romain Calascibetta , Rudi Grinberg and Anil Madhavapeddy . Paper in the proceedings of the OCaml Workshop 2019 . URL (icfp19.sigplan.org) BIB PDF

Programming Unikernels in the Large via Functor Driven Development

2019-05-01T01:00:00Z

Preprint on programming unikernels with ML modules Gabriel Radanne , Thomas Gazagnaire , Anil Madhavapeddy , Jeremy Yallop , Richard Mortier , Hannes Mehnert , Mindy Preston and Dave Scott . Working paper at arXiv . URL (arxiv.org) DOI BIB

Snape: The Dark Art of Handling Heterogeneous Enclaves

2019-03-01T00:00:00Z

Paper on a framework to rearchitect applications for better TEE support at EdgeSys 2019 Zahra Tarkhani , Anil Madhavapeddy and Richard Mortier . Paper in the proceedings of the 2nd International Workshop on Edge Systems, Analytics and Networking . URL (dl.acm.org) DOI BIB PDF

Falsehoods

2019-01-11T00:00:00Z

Being a list of some cool “Falsehoods programmers believe about …” sites, now redundant thanks to https://github.com/kdeldycke/awesome-falsehood : …time …time, more …names …addresses …geography

Distributed Task Scheduling Framework over Irmin

2019-01-01T00:00:00Z

Distributed Task Scheduling Framework over Irmin This is an idea proposed in 2019 as a Cambridge Computer Science Part II project, and has been completed by Mohammed Daudali . Distributed computation and task scheduling frameworks can be decentralised with minimal cost to performance. Furthermore, this decentralisation can provide a significant reduction in the trusted computing base and complexity of the system, affording end consumers a greater level of confidence in the integrity of the results. Moreover, carefully designed persistent and transient data structures can augment this confidence by providing strong isolation guarantees in a multi-tenant system, whilst retaining full transparency over the dynamic data flow graph. This can all be achieved with an API that interfaces directly with...

A strongly consistent index for email using git and MirageOS

2019-01-01T00:00:00Z

A strongly consistent index for email using git and MirageOS This is an idea proposed in 2019 as a Cambridge Computer Science Part II project, and has been completed by Oliver Hope . It was co-supervised with David Allsopp . Maildir is a widely used format for storing emails. Its main benefit is that it uses the filesystem in such a way that client programs do not have to handle locking themselves. The downside of this is that it makes it hard to create a consistent index as we cannot guarantee that the filesystem is in a consistent state when we try to update it. If we did have a consistent index, it would allow for safer concurrent support and the implementation of new features. The aim of this project therefore is to solve the consistency problem. This can be done by using git, the...

Wipeout

2018-12-17T00:00:00Z

I recently decided to go through some old PCs and hard disks (yes, actual spinning bits of metal) and recycle or simply junk them. I figured I should wipe them properly first, and given that they had been installed with a range of OSs, setup a bootable USB stick so that I could boot and wipe in one easy motion. Download Alpine Linux 3.8.1 ISO – I had only x86_64 machines, YMMV obviously Write this to a USB stick using dd (on Linux), or Balena Etcher on Windows Insert the stick and boot the machine after making any necessary BIOS changes If the machine has been unbooted for too long or (as one of mine) has a flat CMOS clock battery so cannot retain time across reboots, set the time manually: date --set="20181217" automatically: setup-alpine to start configuring things, then CTRL-C after...

The OCaml Platform 1.0 with Reason ML

2018-12-11T10:57:40Z

The OCaml Platform 1.0 with Reason ML / Dec 2018 Speaking about the OCaml Platform at the ReasonML meetup hosted by Pusher.

New opam repository layout for large libraries

2018-11-19T00:00:00Z

Managing package manager constraints is getting difficult, particularly given the growth of the number of packages in the opam repository . I'm therefore laying out a new mechanism for the OCaml contributors to submit large package sets, such as those from Jane Street .

A modular foreign function interface

2018-10-01T01:00:00Z

Journal paper on building modular foreign function interfaces Jeremy Yallop , David Sheets and Anil Madhavapeddy . Journal paper in Science of Computer Programming (vol 164). URL (linkinghub.elsevier.com) DOI BIB PDF

Foundations of Computer Science

2018-09-02T01:00:00Z

Here are the various repos used to create the interactive teaching environment we use for 1A Foundations of Computer Science in Cambridge. It may be useful to other professors who are using OCaml in their courses. https://github.com/avsm/teaching-fcs is a private repo, but ping me if are teaching and I'll give you access (it has coursework answers in it). We use a Jupyter notebook, with the course written in Markdown using the mdx OCaml parser which evaluates toplevel phrases through the compiler and promotes the output directly into the markdown. We then convert the Markdown into Jupyter format using a fork of mdx , and then nbconvert it into LaTeX for the printed notes. A JupyterLab installation with a custom OCaml kernel suffices for the live setup. Every student gets their own container on the server and one...

The joys of Dune vendoring

2018-08-15T01:00:00Z

Of the many wonderful features provided by Dune , my personal favourite remains its composability. It doesn’t gain much mention in the manual because, well, there’s not that much to explain about it beyond the simple fact that if you put separate Dune projects in subdirectories underneath your current project, then Dune will use those projects in preference to opam-installed libraries. This allows for a far superior workflow to developing using opam pins, as can be seen from my own recent addition of features to Mirage’s logs-syslog library, which also required an addition to one of its dependencies, syslog-message . Neither of these libraries are presently built using Dune, however porting them both was not particularly difficult, and I thought it would be interesting to...

Bounding data races in space and time

2018-06-01T01:00:00Z

Paper on the OCaml memory model and underlying theory at PLDI 2018 Stephen Dolan , KC Sivaramakrishnan and Anil Madhavapeddy . Paper in the proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation . URL (dl.acm.org) DOI BIB PDF

Urgent problem with camlp5 7.03 and macOS OCaml 4.06.1

2018-05-04T01:00:00Z

This was originally on the posted on the OCaml Platform Blog . Packaging problem with opam-repository camlp5 7.03 when upgrading to OCaml 4.06.1 Between 26 Oct 2017 and 17 Feb 2018, the OPAM package for camlp5 7.03 in opam-repository was under certain circumstances able to trigger rm -rf / on macOS and other systems which don’t by default prevent recursive root deletion. This article contains advice on how to identify if your OPAM installation is affected and what you can do to fix it. TL;DR If rm --preserve-root gives a message along the lines of unrecognised option rather than missing operand and you are running OPAM 1.2.2, ensure you run opam update before upgrading your system compiler to OCaml 4.06.1. If you have already upgraded your system compiler to OCaml 4.06.1 (e.g. with Homebrew) then...

Concurrent System Programming with Effect Handlers

2018-04-01T01:00:00Z

Paper on concurrent systems programming with effect handlers at TFP 2017 Stephen Dolan , Spiros Eliopoulos , Daniel Hillerstrom , Anil Madhavapeddy , KC Sivaramakrishnan and Leo White . Paper in the trends in Functional Programming . URL (link.springer.com) DOI BIB PDF

An architecture for interspatial communication

2018-04-01T01:00:00Z

Paper on the interspatial networking architecture at HotPOST 2018 Anil Madhavapeddy , KC Sivaramakrishnan , Gemma Gordon and Thomas Gazagnaire . Paper in the proceedings of the HotPOST 2018 workshop at the IEEE Conference on Computer Communications . URL (ieeexplore.ieee.org) DOI BIB PDF

Founded Tarides and looking to hire OCaml hackers

2018-03-02T00:00:00Z

I'm delighted to report that I'm helping my long-time collaborator Thomas Gazagnaire to return to his OCaml roots from Docker. He has just founded Tarides, a startup in Paris with the goal of advancing the open source OCaml ecosystem. Founded in Paris in early 2018, Tarides helps developers and companies build secure, performant and resource-efficient network and storage services. We are using MirageOS to run applications without the overhead of a traditional operating system and Irmin to create scalable distributed applications. Tarides offers commercial support and commercial development services for companies interested to run MirageOS or Irmin as part of their technology stack. -- Thomas Gazagnaire

A pain in the num

2018-01-31T00:00:00Z

Fixing a packaging error with num on OCaml 4.06.0 system switches Back in July 2012, the release of OCaml 4.00.0 marked the start of a slow process to remove libraries and tools from the distribution which are not directly related to the compiler and the OCaml Standard Library. For OCaml 4.06.0 it was the turn of the Num library. It is now available as the num package in opam for OCaml 4.06.0 and with a dummy .0 version which is “installed” for prior versions of the compiler which include the library in the distribution. It has always been possible to link with the num library using the num package in ocamlfind , but in order to maintain compatibility with build systems which don’t use ocamlfind, the standalone Num library files are still installed to OCaml’s lib directory, rather...

Probabilistic Programming in OCaml

2018-01-01T00:00:00Z

Probabilistic Programming in OCaml This is an idea proposed in 2018 as a Cambridge Computer Science Part II project, and has been completed by Hari Chandrasekaran . It was co-supervised with Tom Kelly and Liang Wang . With increasing use of machine learning, it is useful to develop frameworks that support rapid development and functional specification of probabilistic models for inference and reasoning. Probabilistic Programming Languages aim to support concise syntax for specifying models and consequently making inference easier. This can pave way to improvements of the model created, more data gathering and further model refinement in an iterative sense. PPL enables easier development of statistical models and allows decoupling inference from modelling. There is a lot of recent work on PPLs,...

Interspatial OS

2018-01-01T00:00:00Z

Interspatial OS Digital infrastructure in modern urban environments is currently very Internet-centric, and involves transmitting data to physically remote environments. The cost for this is data insecurity, high response latency and unpredictable reliability of services. I am working on Osmose -- a new OS architecture that inverts the current model by building an operating system designed to securely connect physical spaces with extremely low latency, high bandwidth local-area computation capabilities and service discovery. In 2018, I was starting to wrap up a multi-year focus on Unikernels , and I went back to look over the state of personal data handling (as I'd finished working on Personal Containers in 2016). Things had regressed fairly dramatically -- central cloud providers and particularly...

CausalRPC: a traceable distributed computation framework

2018-01-01T00:00:00Z

CausalRPC: a traceable distributed computation framework This is an idea proposed in 2018 as a Cambridge Computer Science Part II project, and has been completed by Craig Ferguson . The project aims to implement an RPC framework in OCaml using the Irmin distributed database library as a network substrate. It will explore the trade-offs of a novel data-oriented approach to RPC in which race conditions between clients are resolved automatically by the middleware layer. The core deliverable is a demonstration of an RPC client remotely executing functions with Irmin-serialisable parameters on a server capable of handling concurrent client requests. The project was completed successfully, with an implementation of CausalRPC , a distributed computation framework satisfying the above criteria....

Back to the Future

2017-12-14T00:00:00Z

Ed: the first in a likely series, this is one of the posts I drafted but never published during my seven years in the blogging wilderness I have been having persistent issues on MacOS with Time Machine backups interacting badly with several configuration elements and other services. This is exacerbated by my use of maildirs to backup emails, with my Gmail account leading to a directory with some millions of files. (Yeah, ok, maybe not wise but hey, we are where we are.) So in the course of trying to get backups to work reliably, I found the following commands useful to at least some degree: mdutil -sav to get info about spotlight indexing status sudo mdutil -Ea -i [off|on] to turn off|on spotlight indexing where possible sudo fs_usage -w mdworker mds_stores backupd obtain the current...

Quelle dommage

2017-11-05T00:00:00Z

Ed: this tool is perhaps less relevant now that both Mirage and OPAM have moved on. But perhaps it’ll be resurrected one day so here it is. Largely because I wanted to make a feeble attempt at a French pun, dommage is a tool for Docker containerising Mirage unikernels. From the README : Dommage, Dockerised Mirage dommage is a shell script that wraps the Mirage CLI to make use of Docker containers meaning that: you can cache the OPAM build artefacts in the container image, speeding up local builds; you can re-use the build container image in Travis builds by publishing it, speeding those up considerably; and you can easily test build -t xen targets on OSX. I’ve tried to minimise interference with the normal operation of Mirage CLI so simply replacing mirage with dommage is...

Spring Loading Karabiner in the Autumn

2017-10-30T00:00:00Z

I recently took the plunge and upgraded my OS X. Not to vN of Sierra as I’d hoped, but to v0 High Sierra – the perils of waiting too long… Unfortunately, this toasted 1 my carefully curated keyboard remappings as Karabiner used a kernel extension, for which everything changed. All was not lost however, as the rewrite to support Sierra/High Sierra was well underway. Or so I thought until I realised that the configuration file had changed from XML to JSON. And so my configuration journey began. (But it all ends well, so that’s good.) 1 To be honest, I suspect even the Sierra upgrade would’ve done this. Controlling the config The first thing was to get the new configuration matters under control. I did this per the documentation, symlinking the config subdirectory from my rc-files repo: cd ~...

A bug-fix 12 years in the making

2017-10-30T00:00:00Z

Windows Unicode Support in OCaml 4.06.0 In August 2005, Mantis Report #3771 was opened against OCaml 3.08.4 reporting that Windows applications could not read filenames containing Unicode characters. Various patches have been proposed over the last twelve years leading to GitHub Pull Request #153 almost exactly 2 years ago. This last spring, Clément Franchini at Altair rebased the patch and added tests. Finally, Nicolás Ojeda Bär at LexiFi took on the challenge of completing the patch and minimising the diff with a view to merging in OCaml 4.06.0. The resulting GitHub Pull Request #1200 just made it in time for the feature freeze last month, with a few follow-up pull requests since to polish things up. With luck (and testing!), we have a good working story with the 4.06.0 release candidate, which was pushed yesterday....

Setting up OCaml’s GitHub CI for user forks

2017-09-27T01:00:00Z

There are various Continuous Integration (CI) systems monitoring OCaml. The most publicly visible of these are the two which run on our GitHub repository: Travis , which tests GNU/Linux in 32-bit and 64-bit configurations and also performs a few sanity checks on pull requests, and AppVeyor which tests some of the native Windows ports. OCaml 4.06 was branched on 18 September. One of the problems that we face during the weeks between branching and a release of OCaml is that the increased activity creates huge pressure on our CI systems. This is particularly noticeable on AppVeyor, since the build takes 40–50 minutes and we can only have 1 build running at a time across all pull requests and branches. Travis builds actually take around the same amount of overall time, but they are more...

State of the OCaml Platform

2017-09-08T01:00:00Z

Talk on the state of the OCaml Platform

Windows Console Performance (or lack thereof)

2017-09-06T01:00:00Z

I spent a few days at the weekend refreshing my native Windows fork of Opam and while doing so, found an interesting performance issue for console applications on Windows. One of the jobs porting both Opam and opam-repository for native Windows is updating all the OCaml compiler packages themselves to have Windows build commands. It’s quite extensive, since the instructions have varied across versions and there are also a lot of patches which need to be applied, so this is done using an opam-admin.top script. This little helper program in the Opam developer tools includes a useful function Opam_admin_top.iter_packages which passes every package definition in a repository along with its opam file to a function you provide and writes any changes to that opam back to the repository. There...

The State of the OCaml Platform: Sep 2017

2017-09-01T01:00:00Z

Annual update on the OCaml Pltform at ICFP Anil Madhavapeddy . Paper in the the 7th ACM OCaml Users and Developers Workshop. BIB PDF

Effectively tackling the awkward squad

2017-09-01T01:00:00Z

Paper on how to tackle awkward IO patterns with effect handlers Stephen Dolan , Spiros Eliopoulos , Daniel Hillerstrom , Anil Madhavapeddy , KC Sivaramakrishnan and Leo White . Paper in the the ACM ML Workshop. BIB PDF

Platforms, Packaging, Progress— Addendum

2017-09-01T01:00:00Z

This is a short addendum to my post of a couple of days ago caused by my carelessness in writing the OPAM file . Careful readers will observe the lack of any dependency on notty . Read on for what happened next… The result of this carelessness was that everything worked just fine locally, but my PR to the OPAM package repository failed. Cue much wailing and gnashing of teeth. However, thanks to a moment’s assistance from Daniel Bünzli , this was easy to fix: $ git checkout 0.2.0 # checkout the relevant release version tag $ topkg opam pkg # create the release metadata $ e _build/ocal.0.2.0/opam # invoke editor so I can add the missing dep $ topkg opam submit # submit the updated OPAM metadata, updating the PR Submitting _build/ocal.0.2.0 [ocal-0.2.0.tbz] http://github.com/mor1/ocal/releases/download/0.2.0/ocal-0.2.0.tbz...

Platforms, Packaging, Progress

2017-08-28T01:00:00Z

I recently decided to refresh and update my ocal package, 1 primarily to port it to use the excellent notty before adding support for indicating week-of-year. At the same time, I took the opportunity to update the build infrastructure now that the OCaml world has some shiny new packaging and build tools to go with OPAM , namely topkg and jbuilder . So, starting from Dave Scott’s wiki entry about how to package Mirage libraries, here’s what I had to do… 1 A somewhat over-featured replacement for the standard UNIX cal utility, because I got irritated by its American-centricity and my initial Python replacement was just too slow… Remove Oasis remnants git rm _oasis setup.ml Makefile * _tags myocamlbuild.ml .merlin mv ocal.opam/opam o && git rm - rf ocal.opam && mv o...

ARMing LinuxKit

2017-06-02T01:00:00Z

As some may know, following the Unikernel Systems acquisition, I currently do contract work for Docker Inc. in addition to my day job here at the Cambridge University Computer Laboratory . Recently this has centred on LinuxKit , “ A toolkit for building secure, portable and lean operating systems for containers ” and, specifically, enabling ARM64 support. I’m pleased to say that a basic proof-of-concept is now complete, and we’re working towards getting support merged upstream. The proof-of-concept was developed using the great ARM64 support provided by packet.net , on one of their type 2A boxes. If you fancy trying it out, then hopefully the following instructions will be of use – or just bug me on the packet.net Slack ! Building Start by getting an ARM64 box setup. If you have one to hand, great! If not, you could head over...

Opening discuss.ocaml.org for the community

2017-05-13T01:00:00Z

I opened up a Discourse forum for the OCaml community to use, which is running successfully on https://discuss.ocaml.org. This forum thread collates the feedback and discussions about it.

Docker Docker

2017-05-06T01:00:00Z

Bootstrapping Docker for ARM64 (aka AARCH64) Basic process is: bootstrap ARM64 go toolchain on x86, and install build ARM64 go1.7.5 toolchain needed for docker build bootstrap ARM64 docker v1.10.3 use bootstrapped docker to provide containerised build environment for building later versions Instructions below are for CentOS 7 for Reasons(tm). Package details and so on will vary on other distros. Build go bootstrap toolchain On x86 host: build basic go1.4 sufficient to bootstrap cd curl - O https://storage.googleapis.com/golang/go1.4-bootstrap-20161024.tar.gz tar xzvf go1.4-bootstrap-20161024.tar.gz mv go go1.4 cd go1.4/src ./make.bash cross-compile go1.7 (latest) mkdir - p ~ /go/src cd ~ /go/src git clone https://go.googlesource.com/go cd go git checkout go1.7.5 cd src GOOS = linux GOARCH = arm64 ./bootstrap.bash transfer cross-compiled...

DaLi: Database as a Library

2017-05-01T01:00:00Z

Position paper on building databases-as-a-library at SNAPL 2017 Gowtham Kaki , KC Sivaramakrishnan , Thomas Gazagnaire , Anil Madhavapeddy and Suresh Jagannathan . Paper in the 2nd Summit on Advances in Programming Languages (SNAPL). BIB PDF

Lab GitLab

2017-04-19T01:00:00Z

Recently had cause to do this as part of the SRG’s and OCaml Labs infrastructure. Thought it might be useful to make some notes, so here they are! Assuming your local sys-admin has kindly created you a suitable VM running Ubuntu with login credentials, etc, read on… Note that several commands that follow must be run as root , via use of sudo below. Given that, think twice before just cutting and pasting them in, obviously… And I am not held responsible for anything either way! Install Docker On a new Ubuntu stretch/sid (testing) VM: $ lsb_release - drc Description: Ubuntu 16.04.1 LTS Release: 16.04 Codename: xenial Next, install up-to-date Docker : sudo apt-get install apt-transport-https ca-certificates apt-key adv -- keyserver hkp://p80.pool.sks-keyservers.net:80 \ -- recv-keys...

Google screening

2017-01-29T00:00:00Z

Some time ago, for reasons best known to themselves, a Google recruiter decided to “reach out” on the basis of my GitHub profile to see if I were interested in a role as a Site-Reliability Engineer or possibly a Software Engeering. This entailed a short (~30min) telephone interview to answer some questions. I made a note of those I recalled, in case anyone’s interested. The hawk-eyed and keen-minded among you may discern a certain amount of ambiguity in answers to some of the questions – e.g., is the opposite of malloc() , free() or a garbage collector? are we assuming an Ethernet MAC address? – which the recruiter did not seem to be happy to deal with. But so long as my answer included a reasonable approximation to (presumably) the string they had written down, all was well. What is the Big-O...

Customising Homebrew service

2017-01-04T00:00:00Z

As I could never remember the command to start the offlineimap service using my customised configuration, here it is: brew services start offlineimap ~ /rc-files/homebrew.mxcl.offlineimap.plist

Building Up Your ARMs

2016-11-16T00:00:00Z

Due to the impending finish of the EU FP7 funded User Centric Networking 1 I recently had cause to revisit the excellent work that [Thomas Leonard][talex5] did for the project in getting Xen/ARM running on the [Cubieboard2][cb2] and [Cubietruck][cb3] (aka [Cubieboard3][cb3]). The resulting repo, [mirage/xen-arm-builder][xab], had languished for several months and the past SD card images had some problems and had been allowed to drop off the ’Net as a result. However, sterling work by [Ian Campbell][ijc] at a recent Mirage [hackathon][] had started to resurrect this work based on the [Alpine Linux][alpine] distribution. This seemed a promising place to start, so I did :) Building an Image The end result was an enormous [pull request][pr] that splatted a Brave New World on top of [Thomas’][talex5]...

Debugging with jQuery

2016-11-16T00:00:00Z

I had cause to do this recently, so here’re the runes from http://stackoverflow.com/questions/7474354/include-jquery-in-the-javascript-console for the record: var script = document . createElement ( ' script ' ) ; script . src = " https://ajax.googleapis.com/ajax/libs/jquery/1.6.3/jquery.min.js " ; document . getElementsByTagName ( ' head ' ) [ 0 ] . appendChild ( script ) ;

Zen and the Art of Research Management

2016-11-11T00:00:00Z

I think this is a bit of a classic, the first written form of which I came across in Prof. Roger Needham ’s 50+5 Festschrift celebrating his time at the Cambridge University Computer Lab and Microsoft Research Cambridge . I don’t know who originated it, but the copy there is certainly due to John Naughton and Bob Taylor . I suppose one might quibble point 12, in that I seem to recall Roger did a lot of pacing about, but a good chair is certainly a worthwhile thing to provide. Anyway, I find myself wanting to point at it from time-to-time, so here it is! By John Naughton ( Open University, Milton Keynes, England ), and Robert W. Taylor ( Woodside, California, USA ). HIRE ONLY THE VERY BEST PEOPLE, EVEN IF THEY ARE CUSSED. Perhaps especially if they are cussed. Your guiding principle should be to employ people who are smarter than you. One superb researcher...

Evolving the Unikernel Revolution

2016-11-10T00:00:00Z

I’ve had the pleasure of giving a couple of talks at some fun venues recently, extolling both the virtues of unikernels and talking a bit about where we currently see them as usefully being deployed. Specifically, Operability.io 2016 a couple of weeks ago was enlightening about some of the problems faced in operating production systems. Some great audience questions and follow-ups after the talk, including some who were even wondering when we’ll see unikernels as ready for the desktop! Of course, with the release of the Docker for Mac and [Docker for Windows][dfw] products, it’s arguable that we’ve beaten Linux to that accolade, as both products make extensive use of MirageOS unikernel libraries. Having said that, I was pleased to be told that the message about unikernels having a range of...

Rat Racing Jetlag

2016-11-04T00:00:00Z

As I’ve found myself repeating the same information several times recently, and have to dig out the links in question every time, I figured it’d be useful to write this down once so I can point at it. Jetlag is a first-world problem but can be an annoying one– in recent years I’ve found I have a particular problem getting up in the morning when flying east. So, one day, bored in an airport in the US and with only the entire Internet to hand, I thought I’d look around for an explanation at least. I ended up coming across reports of some physiological research – in rats of course – that suggested a way to avoid jetlag. I’ve since tried it over a dozen times, and found it to work in all cases. At this point I no longer care if it’s placebo affect or a genuine explanation or something else :)...

Unikernels: the rise of the library hypervisor in MirageOS

2016-10-14T01:00:00Z

DockerCon talk on unikernels and MirageOS

Grubbing Around

2016-09-13T01:00:00Z

Nothing earth-shattering here: I recently had the “pleasure” of setting up an ARM64 server. After considerable support, several firmware upgrades, corruption of the main HDD, reinstallation of CentOS7 (recommended, somewhat to my surprise), all that remained was to get an up-to-date Linux built and installed with 32 bit binary support. This took a bit of make config fiddling, but got there after a few tries. And then I had to relearn how grub / grub2 works in this brave new (to me) UEFI CentOS7 world. Herewith some brief commands I found useful while doing so… {% highlight bash %} sudo grep “^menu entry” /boot/efi/EFI/centos/grub.cfg | tr -s “ “ | cut -f 2 -d “’” | cat -n {% endhighlight %} Edit /etc/default/grub to set GRUB_DEFAULT=N for desired value of N Temporarily set the default for...

MirageOS Summer 2016 hack retreat

2016-06-29T01:00:00Z

The functional innards of Docker for Mac and Windows

2016-06-24T01:00:00Z

I gave a talk at the Functional Works meetup, held in Jane Street London about how Docker for Mac and Windows use OCaml and unikernels under the hood .

Ian Eyberg, Joshua Bernstein, Anil Madhavapeddy at OSCON in Austin

2016-06-06T01:00:00Z

Interviewed by The New Stack at OSCON in Austin, Texas

FLICK: Developing and Running Application-Specific Network Services

2016-06-01T01:00:00Z

Paper on application-specific network services at USENIX ATC 2016 Abdul Alim , Richard Clegg , Luo Mai , Lukas Rupprecht , Eric Seckler , Paolo Costa , Peter Pietzuch , Alexander L Wolf , Nik Sultana , Jon Crowcroft , Anil Madhavapeddy , Andrew W. Moore , Richard Mortier , Masoud Koleni , Luis Oviedo , Matteo Miliavacca and Derek McAuley . Paper in the 2016 USENIX Annual Technical Conference (USENIX ATC 16) . URL (usenix.org) BIB PDF

Improving Docker with Unikernels

2016-05-18T01:00:00Z

I announce the release of three big components that form the basis for Docker for Desktop : a hypervisor framework called HyperKit, a networking framework for host translation called VPNKit, and a versioned data management store called DataKit.

Local Knowledge

2016-03-11T00:00:00Z

A relatively long hiatus this time – blame the Cambridge housing market and the need to simultaneously act both incredibly quickly and mind-numbingly slowly. Also obtaining a College Fellowship and experiencing a Company Acquisition were considerable (interesting, welcome) distractions… :) As an interim measure, and because I’ve been asked relatively frequently over the last few months, I thought I’d collect some local knowledge^Wopinion, garnered over many long hard years of hanging about in both Cambridge and Nottingham. What follows should be viewed as indicating nothing more than my lack of imagination. Both Cambridge and Nottingham are blessed with many fine pubs, as well as several truly terrible ones. Less said about the latter the better. But among those that I like, and will occasionally drag...

Declarative Foreign Function Binding Through Generic Programming

2016-02-01T00:00:00Z

Paper on declarative approaches to foreign function bindings at FLOPS 2016 Jeremy Yallop , David Sheets and Anil Madhavapeddy . Paper in the the proceedings of Functional and Logic Programming (FLOPS) . URL (link.springer.com) DOI BIB PDF

Unikernel Systems is now part of Docker

2016-01-21T00:00:00Z

Announced that Unikernel Systems is now part of Docker

Unikernel Systems acquired by Docker

2016-01-21T00:00:00Z

My startup Unikernel Systems was acquired by Docker , and I'll be joining and setting up a UK branch of Docker along with the rest of my team. 'Just like we did with containers, we are interested is democratizing that technology, making it available and useful to the millions of developers and IT pros out there, said Solomon Hykes , founder and chief technology officer for Docker. 'Unikernels allow you to basically get rid of the operating system, and instead compile into the application the small bits of the operating system it really needs.' -- The New Stack You can also see an announcement from me explaining the background story:

Of Mice and Keyboards

2015-11-21T00:00:00Z

A bit of a delay since the last post – lots going on! But anyway: I (relatively) recently upgraded my old skool Macbook Pro (look! built-in Ethernet port! DVD drive!) to El Capitan. This was generally rather less faff that the previous upgrade, though it did seem to take rather more reboots than might have been assumed to be strictly necessary before it settled down, and I’d remembered to fix up permissions for Homebrew with sudo chown -R $(whoami):admin /usr/local . So that was ok. ![Macbook Pro UK Keyboard]({{ site.url }}/images/keyboard-small.png “Macbook Pro UK Keyboard”) Except… I have a slightly odd keyboard and mouse setup. It’s a UK Macbook which means a slightly tweaked keyboard layout compared to the standard US Macbook keyboard. At my desk, I also use a Microsoft Digital Media...

SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems

2015-10-01T01:00:00Z

Paper on formal specificaiton and testing of filesystems at SOSP 2015 Tom Ridge , David Sheets , Thomas Tuerk , Andrea Giugliano , Anil Madhavapeddy and Peter Sewell . Paper in the proceedings of the 25th Symposium on Operating Systems Principles . URL (dl.acm.org) DOI BIB PDF

Personal Data: Thinking Inside the Box

2015-10-01T01:00:00Z

Paper on personal databoxes at the one-in-a-decade Aarhus conference Amir Chaudhry , Jon Crowcroft , Heidi Howard , Anil Madhavapeddy , Richard Mortier , Hamed Haddadi and Derek McAuley . Journal paper in Aarhus Series on Human Centered Computing (vol 1 issue 1). URL (tidsskrift.dk) DOI BIB PDF

Immutable Distributed Infrastructure with Unikernels

2015-09-29T01:00:00Z

Invited talk at NetPL on Immutable Distributed Infrastructure with Unikernels

The State of the OCaml Platform

2015-09-04T01:00:00Z

Update on the state of the OCaml Platform

Not-Quite-So-Broken TLS

2015-08-01T01:00:00Z

Paper on rebuilding TLS securely but practically at USENIX Security 2015 David Kaloper-Mersinjak , Hannes Mehnert , Anil Madhavapeddy and Peter Sewell . Paper in the 24th USENIX Security Symposium (USENIX Security 15) . URL (usenix.org) BIB PDF

Fame and Fortune!

2015-06-24T01:00:00Z

After a slew of HDI related items, a quick MirageOS note. I was invited to give a presentation on MirageOS at the TNG Big TechDay 8 conference. This was an interesting, and very broad based, event. It brought together about 500 people with about 25 speakers over one day, on topics from cognitive science to AI to chess playing algorithms to obviating testing through proof reasoning. And, of course, operating systems and MirageOS ! If you’re interested, the slides used are available at decks.openmirage.org in the usual way, and if you really want to hear me droning on, TNG are making the video (with synchronised slides) available . As ever, comments welcome!

Actual HCI Papers!

2015-06-17T01:00:00Z

I have to confess to being quite pleased to having a couple of HDI -related papers accepted recently (even if we can’t even get proper reviews on the original HDI paper – recently judged out-of-scope for the third time, even though the Special Issue in question seemed bang on target!). The first is a full paper to ECSCW’15 titled Human Data Interaction: Historical Lessons from Social Studies and CSCW . A collaboration with Dr Andy Crabtree , it examines particularly the role of interaction in HDI and explores how past technical approaches, such as Dataware , to the challenges posed by HDI haven’t fully considered the inherently social nature of data. The second is a short paper to the decennial – seriously: every ten years! – Aarhus 2015 conference titled Personal Data: Thinking Inside the Box . This sets out a vision for an embodiment of a Databox : a physical device supported...

Rescuing a Shattered Nexus 4

2015-05-21T01:00:00Z

A little while ago, before I’d done the smart thing and got myself a case for my Nexus 4, I dropped it a couple of inches onto a hard surface at the wrong angle. The screen promptly shattered – and this was bad because without the touch screen, I couldn’t interact with it, I had some photos on it from son#1 birthday party that hadn’t been copied off, and I hadn’t got round to enabling USB access to the filesystem or any of the debug/developer options. So what to do? I really didn’t want to lose those photos. A couple of hours searching the Interwebs and a little bit of experimentation later, and I managed it. Basically, download and apply the clockwork mod bootloader, and this turns on the developer options that allow access to the filesystem via the Android SDK tools. To find out the...

Moving Onto Mirage

2015-05-20T01:00:00Z

For a little while I’ve had this site running as a MirageOS unikernel, shadowing the main site hosted on GitHub . I’ve finally decided to make the switch, as part of moving over to take advantage of Mirage’s DNS and TLS libraries. Following the usual pattern, as previously explained by Amir , Mindy and others, the process is: Construct a static Jekyll site. Write a Travis YAML file to cause Travis to build the unikernel image and commit it back to the deployment repository. Write a Git post-merge hook for the deployment repository, so that the latest unikernel is automatically booted when a merge is detected, i.e., there is a new unikernel image. Write a cron job that periodically polls the deployment repository, pulling any changes. Building a Jekyll...

Unikernels: Functional Infrastructure with Mirage OS

2015-05-12T01:00:00Z

Talk at Esper on functional programming with unikernels

Jitsu: Just-In-Time Summoning of Unikernels

2015-05-04T01:00:00Z

NSDI 2015 talk on Jitsu

Kadupul: Livin' on the Edge with Virtual Currencies and Time-Locked Puzzles

2015-05-01T01:00:00Z

Workshop paper on DIY networking using timelock puzzles Magnus Skjegstad , Anil Madhavapeddy and Jon Crowcroft . Paper in the proceedings of the 2015 Workshop on Do-it-yourself Networking: an Interdisciplinary Approach . URL (dl.acm.org) DOI BIB PDF

Jitsu: Just-In-Time Summoning of Unikernels

2015-05-01T01:00:00Z

Paper on spinning up low-latency unikernels per-connection at NSDI 2015 Anil Madhavapeddy , Thomas Leonard , Magnus Skjegstad , Thomas Gazagnaire , David Sheets , Dave Scott , Richard Mortier , Amir Chaudhry , Balraj Singh , Jon Ludlam , Jon Crowcroft and Ian Leslie . Paper in the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15) . URL (usenix.org) BIB PDF

Diamonds are a Chap's Best Friend

2015-04-09T01:00:00Z

As noted previously , this site is basically a Github -hosted Jekyll site at present, though one that can be built as a Mirage unikernel. Part of the Mirage workflow to publish a new post involves using Travis CI to build and then commit back a new unikernel image. Thus it is currently necessary to run Jekyll in the Travis build scripts, and the dynamism of the Ruby environment meant that this broke (again) recently as one of the github-pages gem’s dependencies now depends on Ruby >= 2.0 while the default Rubies on the Travis Ubuntu image for C language builds is 1.8 (via Ubuntu packaging) or, if you remove that one, 1.9 (via rvm ). Read on to find out how to fix this… The fix that currently works for me turns out to be relatively simple: remove all the...

Taking it to the People People

2015-04-08T01:00:00Z

Looks like I get a chance to run my mouth off again :) Upcoming HDI research seminar , organised by Cambridge Big Data / Digital Humanities . In short, details are: 20th April 2015, 14:00–16:00 in S1, Alison Richard Building, West Road, Cambridge . If you’d like to attend, please do register at http://www.eventbrite.co.uk/e/human-data-interaction-cambridge-big-datadigital-humanities-seminar-tickets-16337148852 . And just because pixels are, in some loose sense, nearly free, here’s the abstract from the seminar link above: The increasing generation and collection of personal data has created a complex ecosystem, often collaborative but sometimes combative, around companies and individuals engaging in the use of these data. We propose that the interactions between these agents warrant a new topic of study: Human-Data Interaction (HDI), that sits at the intersection of various disciplines, including computer...

Reviewing the second year of OCaml Labs in 2014

2015-04-02T01:00:00Z

The OCaml Labs initiative within the Cambridge Computer Laboratory is now just over two years old, and it is time for an update about our activities since the last update at the end of 2013 and 2012 . The theme of our group was not to be pure research, but rather a hybrid group that takes on some of the load of day-to-day OCaml maintenance from INRIA , as well as help grow the wider community and meet our own research agendas around topics such as unikernels . To this end, all of our projects have been highly collaborative, often involving colleagues from OCamlPro , INRIA , Jane Street , Lexifi and Citrix . This post covers our progress in tooling, the compiler and language, community efforts, research projects and concludes with our priorities for 2015. OCaml: it's a dog's life. In...

I Think Someone Else Is Also Following Me...

2015-03-16T00:00:00Z

Turns out others were listening too – notably the USA’s DARPA . The recent announcement of the Brandeis programme makes explicit reference to HDI and our website . This has been picked up by GCN , USAToday , NBCNews , Engadget among others. With $60M potentially on the table, I hope that there’ll be many more who get interested in pushing HDI forwards now :)

I Think Someone's Following Me...

2015-03-02T00:00:00Z

It’s always 1 nice when someone notices what you’re doing, and I was pleasantly surprised recently to find that someone had indeed been watching. 2 1 Well, usually. Unless you’re doing something embarrassing like, e.g., dancing. 2 Though in fact, given the pattern involved, I suspect it may be my colleague, Dr Hamed Haddadi who’s actually being followed. To whit, the fact that they also picked up on a study of third-party web-tracking from last year. Specifically, it appears that someone at MIT Technology Review has noticed the work of me and some of my pals: our work on HDI and a recent sketch of some follow-on work we’re pursuing around building a personal Databox . Independently of that, the latter also happened to get picked up by John Naughton in the Guardian...

ICFP 2015 - a call for sponsorship and how you can help

2015-02-18T00:00:00Z

The call for papers for this year’s International Conference on Functional Programming is about to close in two weeks, and over a hundred cutting-edge research papers will be submitted on the theory, application, and experiences behind functional programming and type theory. In addition to the main conference, there are also over 10 big affiliated workshops that run throughout the week on topics ranging from specific languages ( Erlang , Haskell , OCaml ), the broader commercial community , and even art and music . The ICFP conference experience can be a remarkable one for students. Some great ideas have emerged from random corridor conversations between talks with the likes of Phil Wadler , or from rain-soaked discussions with Simon PJ at Mikeller , or in my case, from being convinced to write a book while in a smoky Tokyo bar. This...

Looping the Loop

2015-02-17T00:00:00Z

In a fit of blogging mania, here’s another one literally barely days after the previous one. Maybe I’ll crack this yet. Anyway, this is just a short one with what verges on a Technical Contribution. To whit: I recently sorted out this domain and was having some issues getting some consistency between what dig , Chrome and my domain provider believed to be the correct state. In particular, I was switching over to make the domain properly live rather than simply a 301 Moved Permanently redirect to my old pages at Nottingham. It turns out this was probably mostly Chrome being confused. It seems that it caches 301 Moved Permanently redirects fairly aggressively and the cached entries are not discarded when you go through the standard mechanisms to clear caches. After a bit of experimentation and...

Talks, Old and New

2015-02-14T00:00:00Z

Thanks to an invitation from Hitesh , I recently got the chance to revisit my old stomping ground at Microsoft Research Cambridge . Well, I say “old” – in the intervening 7 years, they’ve moved to a rather splendid new building at the other end of Cambridge, just next to the station. (And improved the coffee too, not that it wasn’t pretty good to start with!) Anyway, this was a pleasant chance to catch up with some old colleagues, meet some new ones, and even speak to my most recently graduated Ph.D. student, Dr Ewa Luger – and who’d’ve thought that I’d ever end up supervising someone coming from the discipline of Political Science too! The ostensible reason was to talk about the Homework project – a talk I’ve given a few times now – and to lead from that into discussing some of my current agenda, around...

Delivered keynote at BOB 2015 on MirageOS

2015-01-23T00:00:00Z

I hopped over to Berlin to give the keynote at BOB 2015 keynote on functional operating systems. If you're in the region, I highly recommend attending BOB as a superbly organised conference with a diverse and interesting crowd of functional programmers.

Begin, Again!

2015-01-15T00:00:00Z

Specifically, I’ve left Horizon and the School of Computer Science at the University of Nottingham to (re-)join the Cambridge University Computer Laboratory . In celebration, and frankly because it was long overdue anyway, I’ve reworked my website. What do you think? For the curious, or the technically inclined, the site now uses ZURB Foundation 5.5.0 (the current downloadable release as of yesterday), with some slightly customised CSS. The site itself is largely written in Markdown and currently generated using Jekyll to be hosted on Github . It’s actually gone through an interim phase where it was parsed by the OCaml OMD parser before being crunched into a Mirage KV_RO filesystem which is then compiled into a type-safe, self-contained web appliance that serves these pages and no other using the OCaml Cowabloga ,...

Raft Refloated: Do We Have Consensus?

2015-01-01T00:00:00Z

Paper on reproducing the raft consensus protocol Heidi Howard , Malte Schwarzkopf , Anil Madhavapeddy and Jon Crowcroft . Journal paper in ACM SIGOPS Operating Systems Review (vol 49 issue 1). URL (dl.acm.org) DOI BIB PDF

Mergeable persistent data structures

2015-01-01T00:00:00Z

Paper on mergeable data structures using Irmin (nee Irminsule) at JFLA 2015 Benjamin Farinier , Thomas Gazagnaire and Anil Madhavapeddy . Paper in the vingt-sixiemes Journees Francophones des Langages Applicatifs (JFLA 2015). BIB PDF

CUFP'13 scribe's report

2015-01-01T00:00:00Z

Published the scribe's report for CUFP 2013 in JFP Marius Eriksen , Michael Sperber and Anil Madhavapeddy . Journal paper in Journal of Functional Programming (vol 25). URL (cambridge.org) DOI BIB

Codemesh 2014: Nymote: Git Your Own Cloud Here

2014-12-17T00:00:00Z

Gave Codemesh 2014 talk on Nymote

Jitsu: Just-in-Time Summoning of Unikernels (new directions in operating systems)

2014-11-25T00:00:00Z

New Directions in Operating Systems talk on Jitsu

Regional clouds: technical considerations

2014-11-01T00:00:00Z

Report on regional cloud computing law available Jatinder Singh , Jean Bacon , Jon Crowcroft , Anil Madhavapeddy , Thomas Pasquier , W. Kuan Hon and Christopher Millard . Technical report (UCAM-CL-TR-863) at University of Cambridge, Computer Laboratory . URL (cl.cam.ac.uk) DOI BIB PDF

MirageOS 2.0: branch consistency for Xen Stub Domains

2014-10-17T01:00:00Z

At the Xen Summit speaking about branch consistency for Xen Stub Domains

OCaml 2014: The OCaml Platform v1.0

2014-09-05T01:00:00Z

Talk on the OCaml Platform reaching v1.0

Haskell Symposium 2014 Keynote on functional OS design

2014-09-05T01:00:00Z

Gave Haskell Symposium 2014 Keynote on functional OS design

The OCaml Platform v1.0

2014-09-01T01:00:00Z

Paper on the OCaml Platform status Anil Madhavapeddy , Amir Chaudhry , Jeremie Dimino , Thomas Gazagnaire , Louis Gesbert , Thomas Leonard , David Sheets , Mark Shinwell , Leo White and Jeremy Yallop . Paper in the the 4th ACM OCaml Users and Developers Workshop. BIB PDF

Multicore OCaml

2014-09-01T01:00:00Z

First paper on multicore OCaml's design at the OCaml Workshop Stephen Dolan , Leo White and Anil Madhavapeddy . Paper in the the 4th ACM OCaml Users and Developers Workshop. BIB PDF

Irminsule: a branch-consistent distributed library database

2014-09-01T01:00:00Z

Paper at the OCaml Workshop on the Irmin database library Thomas Gazagnaire , Amir Chaudhry , Jon Crowcroft , Anil Madhavapeddy , Richard Mortier , Dave Scott , David Sheets and Gregory Tsipenyuk . Paper in the the 4th ACM OCaml Users and Developers Workshop. BIB PDF

Talks from OCaml Labs during ICFP 2014

2014-08-31T01:00:00Z

It's the ever-exciting week of the International Conference on Functional Programming again in Sweden, and this time OCaml Labs has a variety of talks, tutorials and keynotes to deliver throughout the week. This post summarises all them so you can navigate your way to the right session. Remember that once you register for a particular day at ICFP, you can move between workshops and tutorials as you please. Gothenburg, the location of this year's ICFP conference. Quick links to the below in date order: Talk on Coeffects, a Calculus of Context-dependent Computation , Monday 1st September, 16:30-17:20, ICFP Day 1. Talk on Modular Implicits , Thu 4th September, 14:25-14:50, ML Workshop. Talk on Module Aliases , Thu 4th September, 09:35-10:00, ML Workshop. Talk on Metaprogramming in the Mirage OS , Thu 4th September, 14:50-15:10, ML Workshop. Keynote talk on Unikernels , Fri 5th September, 09:00-10:00,...

FLOSS Weekly 302: Open Mirage

2014-07-23T01:00:00Z

Appeared on FLOSS Weekly 302 about Open Mirage

MirageOS v2.0: a recap of new features

2014-07-22T01:00:00Z

This is a big release for us; after the first version came out earlier in the year, we added in support for ARM devices, a new storage subsystem called Irmin and even a pure OCaml TLS stack.

MirageOS v1.2 released and the runup to 2.0

2014-07-08T01:00:00Z

I announce a point release of MirageOS 1.x, and the exciting run up to the major MirageOS 2.0 release which has lots of new features. The number of Mirage users is growing steadily!

SE Radio Episode 204: Anil Madhavapeddy on the Mirage Cloud Operating System and the OCaml Language

2014-05-01T01:00:00Z

Appeared on SE Radio Episode 204 about Mirage and OCaml

Codio: build Mirage unikernels from a browser

2014-03-26T00:00:00Z

I noticed an offhand tweet from Phil Tomson about Codio adding OPAM support, and naturally had to take a quick look. I was really impressed by the whole process, and ended up building the Mirage Xen website unikernel directly from my web browser in less than a minute, including registration! I signed up to Codio for free (since it’s a public project ) using GitHub oAuth (only public identity access required at first, no repository access). Selected a git project and pointed it at the mirage-www repository. At this point, you get the usual file explorer and code editor view in your browser. The magic begins when you go to “Tools/Terminal”, and an interactive Ubuntu shell pops up. Since Codio added opam support , setting up the Mirage environment is a breeze: I notice Codio supports OCaml and opam on...

Easily OPAM switching to any OCaml feature request

2014-03-25T00:00:00Z

Gabriel Scherer announced an experiment to host OCaml compiler pull requests on GitHub for six months. There is a general feeling that GitHub would be a more modern hosting platform than the venerable but reliable Mantis setup that has in place for over a decade, but the only way to find out for sure is by trying it out for a while. One of the great benefits of using GitHub is their excellent API to easily automate workflows around issues and pull requests. After a suggestion from Jeremy Yallop and David Sheets over lunch, I decided to use this to make it easier to locally apply compiler patches. OPAM has a great compiler switch feature that lets you run simultaneous OCaml installations and swap between them easily. For instance, the default setting gives you access to: $ opam switch system...

ICFP 2014 - a call for sponsorship and how you can help

2014-03-03T00:00:00Z

The call for papers for this year’s International Conference on Functional Programming has just closed, with around a hundred cutting-edge research papers submitted on the theory, application, and experiences behind functional programming. This marks just the beginning of sorting out the program, as there are also over 10 big affiliated workshops that run throughout the week on topics ranging from specific languages ( Erlang , Haskell , OCaml ), the broader commercial community , and even art and music . The ICFP conference experience can be a remarkable one for students. Some great ideas have emerged from random corridor conversations between talks with the likes of Phil Wadler , or from rain-soaked discussions with Simon PJ at Mikeller , or in my case, from being convinced to write a book while in a smoky Tokyo bar. Functional...

Using Dust Clouds to Enhance Anonymous Communication

2014-03-01T00:00:00Z

Paper on building dust clouds for anonymous communication Richard Mortier , Anil Madhavapeddy , Theodore Hong , Derek Murray and Malte Schwarzkopf . Paper in the security Protocols XVIII . URL (link.springer.com) DOI BIB PDF

Unikernels, and the Rise of the Virtual Library Operating System

2014-01-13T00:00:00Z

The Communications of the ACM have just published an article that Dave Scott and I wrote providing a broader background on the concept of Unikernels that we’ve been working on since about 2003, when we started building Melange and the Xen toolstack . You can read either the print article (requires an ACM subscription) or the open access version on the ACM Queue. There's been some interesting discussion about it already online: On Reddit , a number of queries about how it fits into the space of containers, microkernels, and other experimental operating systems. Coverage from eWeek , InfoWorld , and Linux.com , and a couple of interviews on InfoQ covering Mirage and my book on OCaml that give more background on the project. Two of the most interesting bits of feedback for me personally...

Not-quite-so-broken TLS in OCaml

2014-01-01T00:00:00Z

Not-quite-so-broken TLS in OCaml This is an idea proposed in 2014 as a good starter project, and has been completed by Hannes Mehnert and David Kaloper-Mersinjak . It was co-supervised with Peter Sewell . Transport Layer Security (TLS) implementations have a history of security flaws. The immediate causes of these are often programming errors, e.g. in memory manage- ment, but the root causes are more fundamental: the challenges of interpreting the ambiguous prose specification, the complexities inherent in large APIs and code bases, inherently unsafe programming choices, and the impossibility of directly testing conformance between implementations and the specification. This internship was to work on nqsb-TLS, our re-engineered approach to security protocol specification and implementation that addresses...

Consolidating Trust for Client Groups that use TLS to Secure Connections

2014-01-01T00:00:00Z

Consolidating Trust for Client Groups that use TLS to Secure Connections This is an idea proposed in 2014 as a Cambridge Computer Science Part II project, and has been completed by Johann Beleites . It was co-supervised with David Sheets . This project aimed to develop a framework that allows administrators to centrally manage trust in CAs and certificates across a large number of clients. The framework should be responsive and changes in trust should not require any software updates or reboots of client devices. Further, no cooperation from CAs or domain owners should be necessary for a security gain. Performance optimisations should be implemented such that it is usable on a daily basis and this project could integrate with other existing attempts at improving the TLS trust model. Related...

Reviewing the first year of OCaml Labs in 2013

2013-12-29T00:00:00Z

This time last year in 2012, I had just announced the formation of a new group called OCaml Labs in the Cambridge Computer Lab that would combine research and community work towards the practical application of functional programming. An incredible year has absolutely flown by, and I’ve put together this post to summarise what’s gone on, and point to our future directions for 2014. The theme of our group was not to be pure research, but rather a hybrid group that would take on some of the load of day-to-day OCaml maintenance from INRIA , as well as help grow the wider OCaml community. To this end, all of our projects have been highly collaborative, often involving colleagues from OCamlPro , INRIA , Jane Street , Lexifi and Citrix . This post covers progress in tooling , the...

Using Docker to bulk-build OPAM packages on Linux

2013-11-15T00:00:00Z

Now that OCaml 4.01 has been released, there is a frenzy of commit activity in the development trunk of OCaml as the new features for 4.02 are all integrated. These include some enhancements to the type system such as injectivity , module aliases and extension points as a simpler alternative to syntax extensions. The best way to ensure that these all play well together is to test against the ever-growing OPAM package database as early as possible. While we’re working on more elaborate continuous building solutions, it’s far easier if a developer can quickly run a bulk build on their own system. The difficulty with doing this is that you also need to install all the external dependencies (e.g. libraries and header files for bindings) needed by the thousands of packages in OPAM. Enter a hip new lightweight container...

MirageOS and XAPI project update at XenSummit

2013-11-13T00:00:00Z

MirageOS and XAPI project update at XenSummit

Notes from PL and OS 2013 workshop

2013-11-03T00:00:00Z

I co-chaired the Programming Languages and Operating Systems workshop at SOSP 2013, and made livenotes about the (many) papers presented there.

Unikernels: Rise of the Virtual Library Operating System

2013-11-01T00:00:00Z

Article on the Communications of the ACM on unikernels is published Anil Madhavapeddy and Dave Scott . Journal paper in ACM Queue (vol 11 issue 11). URL (dl.acm.org) DOI BIB

Trevi: watering down storage hotspots with cool fountain codes

2013-11-01T00:00:00Z

Paper on fountain coding for datacentre networking at HotNets 2013 George Parisis , Toby Moncaster , Anil Madhavapeddy and Jon Crowcroft . Paper in the proceedings of the Twelfth ACM Workshop on Hot Topics in Networks . URL (dl.acm.org) DOI BIB PDF

First edition of Real World OCaml published

2013-11-01T00:00:00Z

The 1st Edition of Real World OCaml by O'Reilly associates has been released! There have been flurry of signing events, including an upcoming one at OSCON in Austin. Anil Madhavapeddy and Yaron Minsky . Book published by Cambridge University Press . URL (cambridge.org) DOI BIB

Commercial users of functional programming workshop report

2013-11-01T00:00:00Z

Published the scribe's report for CUFP 2012 Michael Sperber and Anil Madhavapeddy . Journal paper in Journal of Functional Programming (vol 23 issue 6). URL (cambridge.org) DOI BIB

A 21st Century IDE

2013-10-13T01:00:00Z

I finally decided to sit down and get the shiny new merlin mode for OCaml working with my emacs configuration. Basically, really rather simple in the end although (in the usual fashion!) I did end up spending considerable time tweaking various other customisations… Most of the information below is based on the following sources: http://github.com/def-lkb/merlin#emacs-interface http://zheng.li/buzzlogs-ocaml/2013/08/23/irc.html http://www.ocamlpro.com/blog/2013/03/18/monthly-03.html Before we begin, install merlin : $ opam install merlin The complete commit change is in my github account (combined with a large cleanup of various other aborted OCaml configurations). Breaking it down a bit, first setup some paths: where to find ocp-indent , merlin.el for merlin-mode , and the ocamlmerlin command itself. Note that this relies on the current state of opam , so when you start emacs be sure to have...

Notes on the first Human Data Interaction workshop

2013-10-02T01:00:00Z

We held the first Human Data Interaction workshop over in Cambridge, with lots of discussion about social networks and the state of play with decentralising them.

OCaml 2013 workshop liveblog

2013-09-24T01:00:00Z

I attended the OCaml 2013 workshop and took live notes of the event. There was a lot going on here, which you can learn more about in the " Reviewing the first year of OCaml Labs in 2013 " roundup as well that I published later in the year.

Liveblogging CUFP 2013

2013-09-22T01:00:00Z

The Commercial Uses of Functional Programming workshop is one of the best industry/academia crossover workshops to attend, and these are my livenotes from the 2013 edition.

OPAM 1.1 beta available, with pretty colours

2013-09-20T01:00:00Z

Thomas Gazagnaire just announced the availability of the OPAM beta release. This has been a huge amount of work for him and Louis , so I’m excited to see this release! Aside from general stability, the main highlights for me are: A switch to the CC0 public-domain-like license for the repository, and LGPL2+linking exception for OPAM itself. The cutover to the new license was the first non-gratuitous use of GitHub’s fancy issue lists I’ve seen, too! As part of this, we’re also beginning a transition over to hosting it at opam.ocaml.org , to underline our committment to maintaining it as an OCaml community resource. Much-improved support for package pinning and updates. This is the feature that makes OPAM work well with MirageOS , since we often need to do development work on a low-level...

The OCaml Platform v0.1

2013-09-01T01:00:00Z

Paper on the OCaml Platform at the OCaml Workshop 2013 Anil Madhavapeddy , Amir Chaudhry , Thomas Gazagnaire , David Sheets , Phillipe Wang , Leo White and Jeremy Yallop . Paper in the the 2nd ACM OCaml Users and Developers Workshop. BIB PDF

Ocamlot: Online OCaml Testing

2013-09-01T01:00:00Z

Presented an OCaml ecosystem testing system David Sheets , Anil Madhavapeddy , Amir Chaudhry and Thomas Gazagnaire . Paper in the the 3rd ACM OCaml Users and Developers Workshop . URL (github.com) BIB PDF

Lost in the Edge: Finding Your Way with DNSSEC Signposts

2013-08-01T01:00:00Z

Paper on DNSSEC-based Signpost servers for better p2p communications at USENIX FOCI Charalampos Rotsos , Heidi Howard , David Sheets , Richard Mortier , Anil Madhavapeddy , Amir Chaudhry and Jon Crowcroft . Paper in the 3rd USENIX Workshop on Free and Open Communications on the Internet (FOCI 13) . URL (usenix.org) BIB PDF

Mirage Developer Preview 1 screencast

2013-07-26T01:00:00Z

Mirage Developer Preview 1 screencast

Grepping the source of every OCaml package in OPAM

2013-04-08T01:00:00Z

A regular question that comes up from OCaml developers is how to use OPAM as a hypothesis testing tool against the known corpus of OCaml source code. In other words: can we quickly and simply run grep over every source archive in OPAM? So that’s the topic of today’s 5 minute blog post: git clone git://github.com/ocaml/opam-repository cd opam-repository opam-admin make cd archives for i in *.tar.gz; \ do tar -zxOf $i | grep caml_stat_alloc_string; \ done In this particular example we’re looking for instances of caml_stat_alloc_string , so just replace that with the regular expression of your choice. The opam-admin tool repacks upstream archives into a straightforward tarball, so you don’t need to worry about all the different archival formats that OPAM supports (such as git or Darcs). It...

Unikernels: library operating systems for the cloud

2013-03-01T00:00:00Z

The first paper on unikernels is pubished at ASPLOS 2013 Anil Madhavapeddy , Richard Mortier , Charalampos Rotsos , Dave Scott , Balraj Singh , Thomas Gazagnaire , Steven Smith , Steven Hand and Jon Crowcroft . Paper in the proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems . URL (dl.acm.org) DOI BIB PDF

Perceived risks of personal data sharing

2013-02-01T00:00:00Z

Paper on dataware computing in the digital economy Anya Skatova , Jaspreet Johal , Robert Houghton , Richard Mortier , Neelam Bhandari , Tom Lodge , Christian Wagner , James Goulding , Jon Crowcroft and Anil Madhavapeddy . Journal paper in Digital Economy: Open Digital. BIB PDF

Concurrent revisions for OCaml

2013-01-01T00:00:00Z

Concurrent revisions for OCaml This is an idea proposed in 2013 as a Cambridge Computer Science Part II project, and has been completed by Dimitar Popov . The biggest challenge when using parallel programming is typically how to keep track of the side effects of computations that are executed in parallel and that involve shared mutable state. Traditional methods for dealing with this issue often limit concurrency, do not provide sufficient determinism and are error prone. Ideally, we would like a concept where all conflicts between parallel tasks are resolved deterministically with minimized effort from the programmer. This project aims to design and build a library for OCaml that implements the concept of concurrent revisions . Concurrent revisions as initially proposed highlight these design...

Evolving TCP: how hard can it be?

2012-12-01T00:00:00Z

Paper on extending TCP in a backwards compatible way at CoNeXT 2013 Zubair Nabi , Toby Moncaster , Anil Madhavapeddy , Steven Hand and Jon Crowcroft . Paper in the proceedings of the 2012 ACM conference on CoNEXT student workshop . URL (dl.acm.org) DOI BIB PDF

Announcing OCaml Labs

2012-10-19T01:00:00Z

I’m very excited to announce OCaml Labs , the latest project to hit the Cambridge Computer Lab. As anyone that hangs out near me probably realises, I very much enjoy functional programming. My weapon of choice tends to be OCaml , as it condenses decades of research into a pragmatic blend of functional, imperative and object-oriented programming styles. What’s perhaps less well known are the steady inroads that OCaml has been making into mission-critical areas of industry. At Jane Street , billions of dollars of transactions are routed through a huge ML code-base that is designed to catch bugs at compile-time . At Citrix , the Xen management toolstack that powers millions of hosts in the cloud is largely written in OCaml . Facebook does sophisticated static analysis using OCaml over their vast PHP codebase to...

Breaking up is easy (with OPAM)

2012-10-17T01:00:00Z

Once the main advantages of having hypervisors is that you can have strongly isolated services within a single machine. But it's really hard to actually build these specialised services; that is, until MirageOS came along. This post discusses how to build so-called "stub domains" for Xen using MirageOS.

OUD 2012: Towards an OCaml Platform and Introducing OCaml Labs

2012-09-17T01:00:00Z

Recording of the OCaml Labs announcement

Building a Xenstore stub domain with MirageOS

2012-09-12T01:00:00Z

Signposts: end-to-end networking in a world of middleboxes

2012-09-01T01:00:00Z

Demoed the Signposts DNSSEC system at SIGCOMM Amir Chaudhry , Anil Madhavapeddy , Charalampos Rotsos , Richard Mortier , Andrius Aucinas , Jon Crowcroft , Sebastian Probst Eide , Steven Hand , Andrew W. Moore and Narseo Vallina-Rodriguez . Journal paper in ACM SIGCOMM Computer Communication Review (vol 42 issue 4). URL (dl.acm.org) DOI BIB

Programming the Xen cloud using OCaml

2012-09-01T01:00:00Z

Paper on programming the Xen cloud using OCaml at the OCaml Workshop Dave Scott , Richard Sharp and Anil Madhavapeddy . Paper in the the 1st ACM OCaml Users and Developers Workshop. BIB PDF

Exploring Compartmentalisation Hypotheses with SOAAP

2012-09-01T01:00:00Z

Paper on control flow analysis to break up applications into compartments Khilan Gudka , Robert M Watson , Steven Hand , Ben Laurie and Anil Madhavapeddy . Paper in the 2012 IEEE Sixth International Conference on Self-Adaptive and Self-Organizing Systems Workshops . URL (ieeexplore.ieee.org) DOI BIB PDF

Cost, Performance & Flexibility in OpenFlow: Pick three

2012-06-01T01:00:00Z

Paper on using MirageOS for better SDN infrastructure with OpenFlow Charalampos Rotsos , Richard Mortier , Anil Madhavapeddy , Balraj Singh and Andrew W. Moore . Paper in the 2012 IEEE International Conference on Communications (ICC) . URL (ieeexplore.ieee.org) DOI BIB PDF

Confidential carbon commuting: exploring a privacy-sensitive architecture for incentivising 'greener' commuting

2012-04-01T01:00:00Z

Paper on our use of data lockers within Cambridge to incentivise more green commuting patterns Chris Elsmore , Anil Madhavapeddy , Ian Leslie and Amir Chaudhry . Paper in the proceedings of the First Workshop on Measurement, Privacy, and Mobility . URL (doi.org) DOI BIB PDF

The case for reconfigurable I/O channels

2012-03-01T00:00:00Z

Paper on a new design for reconfigurable IO that copes with heterogenous software/hardware Steven Smith , Anil Madhavapeddy , Christopher Smowton , Malte Schwarzkopf , Richard Mortier , Robert M Watson and Steven Hand . Paper in the rESoLVE workshop at ASPLOS. BIB PDF

Dreaming of an ARM OCaml

2012-02-25T00:00:00Z

I’ve been meaning to play with Plug Computers for some time now, as I need a low-power embedded system around the house. I recently bought a Soekris Net6501 (a pretty powerful Intel CPU, that even has VT support), but had annoying issues getting it working reliably. I ordered an ARM-based Dreamplug as an alternative (and as a bonus, the Dreamplug is 6x cheaper than the Soekris!). Here are my notes on getting it to work. __LINK_4__ Requirements: Aside from the Dreamplug itself, make sure you order the optional JTAG module. This provides a serial console that is essential to getting any development done with it. I also grabbed the extra 16GB Class 10 SLC SD Card, to act as my home directory. You will also need another functional system running Debian (or a VM on your Mac; whatever is easiest)....

OCaml Labs

2012-01-01T00:00:00Z

OCaml Labs I founded a research group called OCaml Labs at the University of Cambridge, with the goal of pushing OCaml and functional programming forward as a platform, making it a more effective tool for all users (including large-scale industrial deployments), while at the same time growing the appeal of the language, broadening its applicability and popularity. Over a decade, we retrofitted multicore parallelism into the mainline OCaml manager, wrote a popular book on the language, and helped start and grow an OCaml package and tooling ecosystem that is thriving today. Background In my PhD work on Functional Internet Services in around 2003-2007, I developed high performance and reliable protocol implementations in OCaml. Subsequently from 2010, I worked on Personal Containers to build high assurance private...

Macro- and Micro-benchmarking in OCaml

2012-01-01T00:00:00Z

Macro- and Micro-benchmarking in OCaml This is an idea proposed in 2012 as a Cambridge Computer Science Part II project, and has been completed by Sebastian Funk . Benchmarking involves the measurement of statistics such as run-time, memory allocations, garbage collections in a running program in order to analyze its performance and behaviour. To scientifically evaluate and understand the performance of a program, there is often a cycle of: making performance observations about the program finding a potential hypothesis, i.e. a cause for this performance behaviour making predictions on experiments based on this hypothesis comparing the predictions against the actual benchmark results to evaluate the hypothesis. To be able to do all this, there is a need for an effective and robust framework...

CUFP 2011 Workshop Report

2012-01-01T00:00:00Z

Published the scribe's report for CUFP 2011 Anil Madhavapeddy , Yaron Minsky and Marius Eriksen . Journal paper in Journal of Functional Programming (vol 22 issue 1). URL (cambridge.org) DOI BIB

Analysis of the Raft Consensus Protocol

2012-01-01T00:00:00Z

Analysis of the Raft Consensus Protocol This is an idea proposed in 2012 as a Cambridge Computer Science Part II project, and has been completed by Heidi Howard . The Paxos algorithm, despite being synonymous with distributed consensus for a decade, is famously difficult to reason about and implement due to its non-intuitive approach and underspecification. In response, this project aimed to implement and evaluate a framework for constructing fault-tolerant applications, utilising the recently proposed Raft algorithm for distributed consensus. Constructing a simulation framework for our implementation would enable us to evaluate the protocol on everything from understandability and efficiency to correctness and performance in diverse network environments. In retrospect, the complexity of...

OCaml Meeting 2011 - MirageOS

2011-10-19T01:00:00Z

At the OCaml Meeting 2011 speaking about MirageOS

An outing to CUFP 2011 for Mirage

2011-09-29T01:00:00Z

We signed up to do a MirageOS tutorial at ICFP, which is a bit daunting: we had to get all the embedded ARM hardware and laptop support in shape, as well as make it work for a bunch of discerning hackers.

Extending 64-bit MIPS support for LLVM

2011-08-01T01:00:00Z

Extending 64-bit MIPS support for LLVM This is an idea proposed in 2011 as a good starter project, and has been completed by William Morland . It was co-supervised with Robert M Watson . In the summer of 2011, we hosted William Morland to do an internship in the Computer Lab just as the CTSRD/CHERI project kicked off. I was interested in MIPS as a potential target for MirageOS, and Robert M Watson in using it for the future CHERI processor. William Morland hacked on the gxemul MIPS simulator, validating (and often creating) the CHERI test suite against the gxemul simulator. He then shifted gears into the (then experimental) LLVM/MIPS backend, filling in missing instructions and finding bugs via exercising the test suite. His LLVM repository is up at GitHub , along with the discussions from back in 2011 on the...

Control flow analysis for privilege separation

2011-08-01T01:00:00Z

Control flow analysis for privilege separation This is an idea proposed in 2011 as a good starter project, and has been completed by Chris Harding and Ross McIlroy . It was co-supervised with Robert M Watson . In the summer of 2011, we hosted Chris Harding and Ross McIlroy to do an internship in the Computer Lab working just as the CTSRD/SOAPP project kicked off. Ross McIlroy built a tool called privgrind , using valgrind that tracks, for all data addresses touched, the list of functions that wrote or read from the address and how much they wrote or read. Chris Harding then built a visualiser for this that output the complex control flow graph that results from this as a privsep-visualiser which would then form a guideline for future compartmentalisation activities. CFG of OpenBSD's syslogd The results of this work...

Dynamics for ML using Meta-Programming

2011-07-01T01:00:00Z

Published dyntype at the Workshop on Generative Technologies Thomas Gazagnaire and Anil Madhavapeddy . Journal paper in Electronic Notes in Theoretical Computer Science (vol 264 issue 5). URL (linkinghub.elsevier.com) DOI BIB PDF

Dynamics for ML using Meta-Programming

2011-07-01T01:00:00Z

Paper on statically typed value persistence for OCaml in ENTCS 2011 Thomas Gazagnaire and Anil Madhavapeddy . Journal paper in Electronic Notes in Theoretical Computer Science (vol 264 issue 5). URL (linkinghub.elsevier.com) DOI BIB PDF

Delimited continuations vs Lwt for threads

2011-06-18T01:00:00Z

DataCaml: distributed dataflow programming in OCaml

2011-06-11T01:00:00Z

Distributed programming frameworks like Hadoop and Dryad are popular for performing computation over large amounts of data. The reason is programmer convenience: they accept a query expressed in a simple form such as MapReduce , and automatically take care of distributing computation to multiple hosts, ensuring the data is available at all nodes that need it, and dealing with host failures and stragglers. A major limitation of Hadoop and Dryad is that they are not well-suited to expressing iterative algorithms or dynamic programming problems. These are very commonly found patterns in many algorithms, such as k-means clustering , binomial options pricing or Smith Waterman for sequence alignment. Over in the SRG in Cambridge, we developed a Turing-powerful distributed execution engine called CIEL that addresses...

Reconfigurable Data Processing for Clouds

2011-05-01T01:00:00Z

Paper on what a Xen+FPGA cloud would look like at FCCM Anil Madhavapeddy and Satnam Singh . Paper in the 2011 IEEE 19th Annual International Symposium on Field-Programmable Custom Computing Machines . URL (ieeexplore.ieee.org) DOI BIB PDF

Functional programming gone wild in the SRG

2011-04-18T01:00:00Z

We've been doing loads of OCaml programming in the Systems Research Group, and this blog post lays out some of the things going on. It ranges from OCaml hacking, over to the CIEL distributed execution engine, and even some Haskell hacking ongoing for distributed execution.

Camel Spotting in Paris

2011-04-15T01:00:00Z

I'm at the 2011 OCaml Users Group in Paris, reporting on some splendid talks this year. It looked like around 60-70 people in the room, and I had the pleasure of meeting users all the way from Russia to New York as well as all the Europeans! Js_of_ocaml First up was Pierre Chambart talking about the js_of_ocaml compiler. It compiles OCaml bytecode directly to Javascript, with few external dependencies. Since the bytecode format changes very rarely, it is simpler to maintain than alternatives (such as Jake Donham’s ocamljs ) that require patching the compiler tool-chain. Javascript objects are mapped to dynamic OCaml objects via a light-weight ## operator, so you can simply write code like: class type window = object method alert : js_string t -> unit meth method name : js_string t prop end let...

CIEL: A universal execution engine for distributed data-flow computing

2011-03-01T00:00:00Z

Paper on CIEL, a distributed dataflow engine, at USENIX NSDI 2011 Derek G Murray , Malte Schwarzkopf , Christopher Smowton , Steven Smith , Anil Madhavapeddy and Steven Hand . Paper in the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11) . URL (usenix.org) BIB PDF

Unclouded vision

2011-01-01T00:00:00Z

Paper on a vision for a semi-federated cloud for personal data at ICDCN Jon Crowcroft , Anil Madhavapeddy , Malte Schwarzkopf , Theodore Hong and Richard Mortier . Paper in the proceedings of the 12th International Conference on Distributed Computing and Networking. BIB PDF

Simulating XMPP Group Communication

2011-01-01T00:00:00Z

Simulating XMPP Group Communication This is an idea proposed in 2011 as a Cambridge Computer Science Part II project, and has been completed by Farhān Mannān . The problem of getting a digital message from one place to another has a staggering range of possible scenarios, constraints and applications. Humans and devices are in constant dialogue, with various constraints and contracts being invisibly maintained. Even the most flippant instant message sets layers of protocols in motion, all straining to resolve identities and propagate information transparently across disparate physical components that must present a logically unified front to users. Subtleties like authentication, encryption and anonymity abound. This project aims to build an OCaml-based simulator (using the ocamlgraph...

Functional ABNF parser generators

2011-01-01T00:00:00Z

Functional ABNF parser generators This is an idea proposed in 2011 as a Cambridge Computer Science Part II project, and has been completed by Nicholas Skehin . Writing internet servers is a difficult proposition. On some levels it seems as though we haven’t made much progress since the 1970s, as popular servers such as Apache and nginx for HTTP, BIND for DNS and qmail for IMAP for many Internet protocols still tend to be written in C. While it is not impossible to write robust software in C, it does tend to be extremely difficult and almost all of the above have suffered from their fair share of security vulnerabilities. With the advent of higher level programming languages, this does not need to be the case any longer. Modern functional languages such as OCaml and Haskell can be competitive...

Building the Xen toolstack using OCaml

2010-11-05T00:00:00Z

Talk on building the Xen toolstack using OCaml

Mirage: A New Multi-Scale Operating System for Clouds and Crowds (2014)

2010-10-25T01:00:00Z

At LinkedIn giving tech talk about Mirage

Self-hosting MirageOS website

2010-10-11T01:00:00Z

I managed to get early MirageOS suitably feature-complete enough that we could run the Mirage website using Mirage. This was all very satisfying after hacking on the TCP/IP stack for ages.

The personal container, or your life in bits

2010-10-01T01:00:00Z

Paper on personal containers for data management at the UK Digital Economy meeting Richard Mortier , Chris Greenhalgh , Derek McAuley , Alexa Spence , Anil Madhavapeddy , Jon Crowcroft and Steven Hand . Journal paper in Digital Futures (vol 10). URL (mort.io) BIB PDF

Using functional programming within an industrial product group: perspectives and perceptions

2010-09-01T01:00:00Z

Paper on our experiences with writing the Xen control stack in OCaml at ICFP 2010 Dave Scott , Richard Sharp , Thomas Gazagnaire and Anil Madhavapeddy . Paper in the proceedings of the 15th ACM SIGPLAN international conference on Functional programming . URL (dl.acm.org) DOI BIB PDF

Turning Down the LAMP: Software Specialisation for the Cloud

2010-06-22T01:00:00Z

At HotCloud for the first talk about MirageOS

Turning Down the LAMP: Software Specialisation for the Cloud

2010-06-01T01:00:00Z

Workshop paper on the early MirageOS architecture and evaluation at HotCloud 2010 Anil Madhavapeddy , Richard Mortier , Ripduman Sohan , Thomas Gazagnaire , Steven Hand , Tim Deegan , Derek McAuley and Jon Crowcroft . Paper in the 2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 10) . URL (usenix.org) BIB PDF

Creating high-performance, statically type-safe network applications

2010-05-01T01:00:00Z

My PhD thesis is now also published as a print book Anil Madhavapeddy . Technical report (UCAM-CL-TR-775) at University of Cambridge, Computer Laboratory . URL (cl.cam.ac.uk) DOI BIB PDF

Yurts for Digital Nomads

2010-04-29T01:00:00Z

The App Engine data collector for Personal Containers is coming on nicely, and is on track for an alpha preview release fairly soon . Working with AppEngine has been interesting; it’s got excellent availability and you can’t beat the price (free), but coding robust Python that doesn’t trip over the tight resource limits for individual requests, asynchronous tasks and queries is tricky. While it is good for small records such as my iPhone or Find My iPhone GPS traces traces, it doesn’t work so well with my gigabytes of photographs or decades of e-mail. This confirmed our earlier intuition that there is no one perfect solution for personal data handling; instead, we need to embrace diversity and construct an infrastructure that can cope with change over the coming decades. Mobile...

Pulling together a user interface

2010-04-15T01:00:00Z

We’ve been hacking away on fleshing out the App Engine node for personal containers. We’re building this node first because, crucially, deploying an App Engine VM is free to anyone with a Google account. The service itself is limited since you can only respond to HTTP or XMPP requests and do HTTP fetches, and so its primary use is as an always-on data collection service with a webmail-style UI written using extjs . Personal containers gather data from a wide variety of sources, and normalise them into a format which understands people (address book entries, with a set of services such as e-mail, phone, IM and online IDs), places (GPS, WOEID), media (photos, movies) and messages (Tweets, emails, Facebook messages). I’ll post more about the data model behind personal containers in a...

Multiscale not multicore: efficient heterogeneous cloud computing

2010-04-01T01:00:00Z

Paper on our vision for multiscale programming at the BCS Visions 2010 conference Anil Madhavapeddy , Richard Mortier , Jon Crowcroft and Steven Hand . Paper in the proceedings of the 2010 ACM-BCS Visions of Computer Science Conference . URL (dl.acm.org) BIB PDF

Opening a website

2010-03-29T01:00:00Z

We've been working away at building a new type of database to help individuals keep reigns on their ever-increasing personal digital information. The first prototypes run freely on Google App Engine to gather your data behind-the-scenes, and we are working on more advanced versions that run on embedded devices and the cloud. If you’re interested in keeping track of your personal data, you can start off with the installation instructions to clone your own version. After that, read up on the design of the system (which is still changing as we research new ideas around it). When you find something you want to fix, or add a new plugin data source, just clone the code and send us back fixes!

Privacy Butler: A Personal Privacy Rights Manager for Online Presence

2010-03-01T00:00:00Z

Paper on privacy butler services for more private data management Ryan Wishart , Domenico Corapi and Morris Sloman . Paper in the 2010 8th IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops) . URL (ieeexplore.ieee.org) DOI BIB PDF

Creating high-performance, statically type-safe network applications

2010-03-01T00:00:00Z

PhD thesis now available as a technical report Anil Madhavapeddy . Technical report (UCAM-CL-TR-775) at University of Cambridge, Computer Laboratory . URL (cl.cam.ac.uk) DOI BIB PDF

Unikernels

2010-01-01T00:00:00Z

Unikernels I proposed the concept of "unikernels" -- single-purpose appliances that are compile-time specialised into standalone bootable kernels, and sealed against modification when deployed to a cloud platform. In return they offer significant reduction in image sizes, improved efficiency and security, and reduce operational costs. I also co-founded the MirageOS project which is one of the first complete unikernel frameworks, and also integrated them to create the Docker for Desktop apps that are used by hundreds of millions of users daily. While working on Personal Containers in late 2008, I had a need to run lots of distributed edge nodes holding personal data. The state of computer security is generally a disaster when it comes to leaving software unupgraded for even a few months, so...

Functional Reactive Web Applications

2010-01-01T00:00:00Z

Functional Reactive Web Applications This is an idea proposed in 2010 as a Cambridge Computer Science Part II project, and has been completed by Henry Hughes . There are a variety of programming languages which can be used to create desktop applications, and each provides different tradeoffs. This could be anything from the runtime guarantees the programming language provides to rapid development and prototyping. It does not make much difference to the user which of these languages was used, as all they want to do is run their favourite application reliably. When writing an application for the web, however, the programmer is forced to use a specific set of APIs that come under the umbrella term AJAX (Asynchronous JavaScript and XML). AJAX involves writing client-side code in JavaScript and...

Combining Static Model Checking with Dynamic Enforcement Using the Statecall Policy Language

2009-11-01T00:00:00Z

Paper on a DSL for specifying temporal protocol automata at ICFEM 2009 Anil Madhavapeddy . Paper in the formal Methods and Software Engineering . URL (link.springer.com) DOI BIB PDF

Personal Containers

2009-01-01T00:00:00Z

Personal Containers As cloud computing empowered the creation of vast data silos, I investigated how decentralised technologies might be deployed to allow individuals more vertical control over their own data. Personal containers was the prototype we built to learn how to stem the flow of our information out to the ad-driven social tarpits. We also deployed personal containers in an experimental data locker system at the University of Cambridge in order to incentivise lower-carbon travel schemes. I've had a passion for self-hosted, decentralised computing for many years since Nick Ludlam and I set up the recoil.org collective in the late 90s. In late 2008, I'd been working on early cloud computing as part of the Xen Hypervisor project and already seeing the rapid rise of centralised data...

Peeking under the hood of High Availability

2008-09-17T01:00:00Z

Well, the big launch of XenServer 5 has gone smoothly, and with it have arrived a flood of questions about how exactly the new High Availability functionality works. I’ll use this post to explain the overall architecture of HA in XenServer 5, and also how some of the fault detection and failure planning works. Fundamentally, HA is about making sure important VMs are always running on a resource pool. There are two aspects to this: reliably detecting host failure , and computing a failure plan to deal with swift recovery. Detecting host failure reliably is difficult since you need to remotely distinguish between a host disappearing for a while versus exploding in a ball of flames. If we mistakenly decide that a master host has broken down and elect a new master in its place, there may be...

Shedding light on XenApp on XenServer performance tuning

2008-08-04T01:00:00Z

You won’t be surprised to hear that we spend a lot of time improving XenApp performance when running on XenServer . Although there are some good benchmark comparisons available (such as the Tolly Group report), I still get a lot of customers asking about what the “secret sauce” is. I sat down with George Dunlap, the lead XenServer performance engineer to chat about the very first optimisation we did back in XenServer 4.0 last year. Before we dive in, we first need to explain how a normal operating system handles memory. George explains: Modern desktop and server processors don’t access memory directly using its physical address. They use ‘ virtual memory ’ to separate the addresses that processes use to read and write memory from the actual memory itself. This allows operating systems to...

Installing Ubuntu on XenServer

2008-07-02T01:00:00Z

I thought I’d kick off my Citrix blog with a question I get pretty often from Linux enthusiasts: how to install unsupported Linux distributions on XenServer 4.1. The most common solution people find is to use the "Other Install Media" template, insert the distribution installation CD, and find that the mouse cursor doesn’t work when they boot into X11. The reason for this is that they are using the hardware-assisted emulation mode of installing Linux. In this mode (dubbed “HVM”), all input and output is emulated, and in particular the mouse interface uses the USB tablet interface. If the distribution doesn’t include a driver for USB tablets, then no mouse will appear. Windows guests run at high-speed in HVM mode due to the installation of the XenServer tools which install high-speed drivers ,...

Enhancing web browsing security on public terminals using mobile composition

2008-06-01T01:00:00Z

Paper on splitting trust between smartphones and webrowsers at MobiSys 2008 Richard Sharp , Anil Madhavapeddy , Roy Want and Trevor Pering . Paper in the proceedings of the 6th international conference on Mobile systems, applications, and services . URL (dl.acm.org) DOI BIB PDF

Melange: creating a "functional" internet

2007-06-01T01:00:00Z

Won best student paper for my PhD work on a high-performance functional packet parsing DSL at Eurosys 2007! Anil Madhavapeddy , Alex Ho , Tim Deegan , Dave Scott and Ripduman Sohan . Journal paper in ACM SIGOPS Operating Systems Review (vol 41 issue 3). URL (dl.acm.org) DOI BIB PDF

Interacting with mobile services: an evaluation of camera-phones and visual tags

2007-02-01T00:00:00Z

Journal paper on interacting with mobile services using camera-phones Eleanor Toye Scott , Richard Sharp , Anil Madhavapeddy , Dave Scott , Eben Upton and Alan Blackwell . Journal paper in Personal and Ubiquitous Computing (vol 11 issue 2). URL (doi.org) DOI BIB PDF

Fighting Crimeware: An Architecture for Split-Trust Web Applications

2006-04-01T01:00:00Z

New paper Fighting Crimeware: An Architecture for Split-Trust Web Applications available Richard Sharp , Anil Madhavapeddy , Roy Want , Trevor Pering and John Light . Technical report (IRC-TR-06-053) at Intel Research. BIB PDF

SPLAT: A Tool for Model-Checking and Dynamically-Enforcing Abstractions

2005-08-01T01:00:00Z

Workshop paper on temporal automata for protocol specifications at SPIN 2005 Anil Madhavapeddy , Dave Scott and Richard Sharp . Paper in the model Checking Software . URL (link.springer.com) DOI BIB PDF

Ubiquitious Computing needs to catch up with Ubiquitous Media

2005-07-01T01:00:00Z

Position paper on ubiquitous computing approaches to emerging stream media appliances Richard Sharp and Kasim Rehman . Journal paper in IEEE Pervasive Computing (vol 4 issue 3). URL (ieeexplore.ieee.org) DOI BIB PDF

Audio networking: the forgotten wireless technology

2005-07-01T01:00:00Z

New paper Audio networking: the forgotten wireless technology available Anil Madhavapeddy , Richard Sharp , Dave Scott and Alastair Tse . Journal paper in IEEE Pervasive Computing (vol 4 issue 3). URL (ieeexplore.ieee.org) DOI BIB PDF

A Study of Bluetooth Propagation Using Accurate Indoor Location Mapping

2005-07-01T01:00:00Z

Ubicomp paper on a study of indoor bluetooth propagation using the Active Bat system Anil Madhavapeddy and Alastair Tse . Paper in the ubiComp 2005: Ubiquitous Computing. DOI BIB PDF

OpenBSD C2K5 thoughts

2005-06-04T01:00:00Z

Finally had some time to get back from the OpenBSD hackathon and take stock of what I worked on. It was pretty interesting one this year, as I went without having much idea of what to work on (unlike last year, when I had a mad backlog to catch up on). Some stuff I did during the week included: Clean up the atomicio interface used in OpenSSH and nc(1) to provide simpler semantics. Error checking from read/write functions are a real headache in C, as the functions return -1 on error, which means a signed ssize_t is returned. However, they accept an unsigned value as the size of the buffer to process, which means they could potentially return a value outside the range of the return value. This means you have to check if the return is -1 , which indicates an error, and otherwise...

On the challenge of delivering high-performance, dependable, model-checked internet servers

2005-06-01T01:00:00Z

Paper on temporal automata for protocol implementations at HotDep 2005 Anil Madhavapeddy and Dave Scott . Paper in the proceedings of the First Conference on Hot Topics in System Dependability . URL (dl.acm.org) BIB PDF

Using visual tags to bypass Bluetooth device discovery

2005-01-01T00:00:00Z

While designing Spotcodes , we realised that visual tags are a much better mechanism to advertise security keys to users instead of the error prone and much more difficult to use Bluetooth device discovery protocol. We duly implemented the direct system, and conducted a user study with Eleanor Toye Scott who was down the corridor working with the HCI group in the Computer Lab. The resulting journal paper on our SpotCode visual tag login system is a fun blend of systems and human factors. We do not believe that printed tags are competing with RFID and NFC for the prize of "universally accepted connection establishment technology". Instead we observe that each offers complementary tradeoffs in terms of cost, data capacity, interaction distance, client-device compatibility and visibility. We...

Using smart phones to access site-specific services

2005-01-01T00:00:00Z

Article on using cameraphones to access site-specific services in IEEE Pervasive Computing Eleanor Toye Scott , Richard Sharp , Anil Madhavapeddy and Dave Scott . Journal paper in IEEE Pervasive Computing (vol 4 issue 2). URL (ieeexplore.ieee.org) DOI BIB PDF

The Broadband Phone Network: Experiences with Context-Aware Telephony

2005-01-01T00:00:00Z

Report on our hacking on the AT&T Broadband Phone Ripduman Sohan , Alastair Tse and Anil Madhavapeddy . Technical report (CUED/F INFENG/TR.533) at Cambridge University Engineering Department . URL (cam-orl.co.uk) BIB PDF

Using camera-phones to interact with context-aware mobile services

2004-12-01T00:00:00Z

A technical report is now available on our SpotCode visual tag system, and includes a user study lead by Eleanor Toye Scott which tested its benefits against conventional mobile interfaces. Eleanor Toye Scott , Anil Madhavapeddy , Richard Sharp , Dave Scott , Alan Blackwell and Eben Upton . Technical report (UCAM-CL-TR-609) at University of Cambridge, Computer Laboratory . URL (cl.cam.ac.uk) DOI BIB PDF

Connecting Paper and Online Worlds by Cellphone

2004-10-07T01:00:00Z

In what is definitely our most exciting media coverage yet, Spotcodes are featured in the New York Times! When you think of a public information kiosk, your mental picture might include greasy touch screens, broken trackballs and frozen monitors. But researchers at an Intel-financed lab at Cambridge University have developed a way to replace displays like those with something portable, not to mention personal: a cellphone's built-in camera and screen. They and others plan to use commercially available hardware to turn the camera-equipped cellphone into a mouse, remote control, keyboard and more. -- New York Times Richard Sharp got cited as I wasn't in the department that day when the journalist showed up at Intel Research! "Instead of having all the hassle of putting things out in the...

Using Camera-Phones to Enhance Human-Computer Interaction

2004-09-01T01:00:00Z

We gave a demo at UbiComp 2004 all the way in Tokyo on our SpotCode visual tag system. It went very well, including some time to do some sightseeing in Japan and visit the sumo wrestling championships! Anil Madhavapeddy , Dave Scott , Richard Sharp and Eben Upton . Paper in the adjunct Proceedings of Ubicomp 2004 (Demo Track) . URL (ubicomp.org) BIB PDF

MIT Technology review covers SpotCodes

2004-09-01T01:00:00Z

We got more coverage of SpotCodes and our startup High Energy Magic , leading to lots of interest in the technology. Public touch-screen displays such as airport check-in kiosks aren’t known for having versatile interfaces; they usually lack keyboards or pointing devices, limiting users to a few navigational buttons. But new software from High Energy Magic of Cambridge, England, turns a camera phone with a Bluetooth wireless connection into a portable mouse and keyboard that can take full command of public displays, doing away with the old touch screen. Working with Intel’s Cambridge research lab, High Energy Magic has developed a set of circular symbols, similar in concept to bar codes, that can be displayed by public terminals. Camera phones loaded with the company’s software can translate...

Exploring tradeoffs in location-aware gaming using smartphones

2004-08-01T01:00:00Z

The summer of 2004 was sufficient full of procrastination that the members of the SN17 collective in the Computer Lab decided to build a computer game. But it wasn't enough to just play the game on our phones -- instead, we combined all the public displays in the corridors, and then added in a cutting-edge 5cm-accurate ActiveBAT , and built a Symbian-based Capture The Flag game where we all had to run around and tag each other physically while tracking the flag virtually. Was it mad? Yes. Was it fun? Yes. Did it get us a paper into the SIGCOMM NetGames workshop? Yes! Our novel contributions include: (i) creating a fast-paced, close quarters, location-aware game, (ii) exploring the tradeoffs between the accuracy of a location system, the I/O capabilities of current mobile hardware, and the...

From the prawn of time

2004-06-07T01:00:00Z

I gave a talk at NotCon 2004 on SpotCodes, and it got covered in Wired Magazine! Of course, I shared the stage with a man telling the time via a prawn sandwich , so the limelight wasn't all just mine... Anil Madhavapeddy and his colleagues at High Energy Magic think camera phones should be used for more than taking bad pictures. The company's SpotCode reader software lets camera phones recognize a circular tag and then communicate via Bluetooth with a local server. -- Roxanne Khamsi for Wired

Cellphone as a virtual mouse/keyboard

2004-05-27T01:00:00Z

The SpotCode cellphone software was spun out into a startup called High Energy Magic Ltd, and was covered on Slashdot. Check this out! High Energy Magic have announced a public beta of software to let you use your camera-phone as a physical mouse by just pointing and clicking and rotating it in the air. There were also articles on DeviceForge that were picked up by quite a few outlets. Update: You can see some of the videos under Ubiquitous Interaction Devices as well.

Context-Aware Computing with Sound

2003-10-01T01:00:00Z

While working as an intern at Intel Research Cambridge, Dave Scott and Richard Sharp and I put together a fun system based on the emerging new class of smartphones. The project kicked off when we randomly experimented with our fancy Nokia smartphones and discovered that they didn't have anti-aliasing filters on the microphones! We argued that [...] audio networking can be used as the basis for developing context-aware applications. Audio networking allows standard devices fitted with speakers and microphones (e.g. PDAs, laptops, desktop PCs and mobile phones) to exchange data and infer information about their environment. One of the key advantages of audio networking is that it enables context-aware applications to be immediately deployed on a large scale without requiring users to purchase...

My static C bounds checker extension merged into OpenBSD

2003-06-27T01:00:00Z

After many rounds of review and helpful feedback from fellow developers, I merged my GCC static bounds checking extension into OpenBSD today! Introduce a simple static checker for making sure that the bounds length passed to common functions such as strlcpy/strlcat match the real length of the buffer. It also checks to make sure that the bound length was not incorrectly derived from a sizeof(pointer) operation. Functions must be marked with the new attribute bounded , and warnings are turned on by -Wbounded. Specifying -Wformat also enables bounds checking for scanf(3) bounds to '%s' format variables. -Wall now turns on -Wbounded also. The checking is pretty limited right now to constant parameters, and the buffers must be statically declared, and not inside a record type. This simple checking still found hundreds...

The Case for Abstracting Security Policies

2003-06-01T01:00:00Z

My first ever academic paper, written with the expert guidance of Alan Mycroft and my PhD colleagues Dave Scott and Richard Sharp ! We worked on a system call policy language to help constrain application access to privileged resources, and implemented this on OpenBSD using systrace . The paper describing the declarative language was presented at SAM 2003 in Las Vegas. "Untrusted code" is just as much a social problem as it is a technical problem. Looking for a complete solution is unrealistic: it is analogous to looking for a solution to crime in general. With this in mind, we do not claim that our proposed framework is a panacea. However, although a number of security problems remain (e.g. covert channel leakage), we claim that our system offers the potential to raise the security level of...

Moving to anil.recoil.org

2003-05-16T01:00:00Z

I've taken the opportunity to redesign my homepage and switch to its hopefully-permanent URL on anil.recoil.org . Many thanks to Jon Parise for giving me permission to base my HTML upon his homepage's, saving me lots of design trouble!

Xen 2002

2003-01-04T00:00:00Z

The first technical report on the Xen Hypervisor hypervisor is now available. I mainly contributed to the early NetBSD port (but have run into a snag with the lack of linear page tables in our paravirtual page implementation). Paul R. Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Evangelos Kotsovinos , Anil Madhavapeddy , Rolf Neugebauer , Ian Pratt and Andrew Warfield . Technical report (UCAM-CL-TR-553) at University of Cambridge, Computer Laboratory . URL (cl.cam.ac.uk) DOI BIB

Ubiquitous Interaction Devices

2003-01-01T00:00:00Z

Ubiquitous Interaction Devices I investigated how to interface the new emerging class of smartphone devices (circa 2002) with concepts from ubiquitous computing such as location-aware interfaces or context-aware computing. I discovered the surprisingly positive benefits of piggybacking on simple communications medium such as audible sound and visual tags. Our implementations of some of these ended up with new audio ringtone and visual smart tags that worked on the widely deployed mobile phones of the era. In 2003, the mobile phone market had grown tremendously and given the average consumer access to cheap, small, low-powered and constantly networked devices that they could reliably carry around. Similarly, laptop computers and PDAs became a common accessory for businesses to equip their...

Functional Internet Services

2003-01-01T00:00:00Z

Functional Internet Services My PhD dissertation work proposed an architecture for constructing new implementations of standard Internet protocols with integrated formal methods such as model checking and functional programming that were then not used in deployed servers. A more informal summary is "rewrite all the things in OCaml from C!", which lead to a merry adventure into implementing many networks protocols from scratch in a functional style, and learning lots about how to enforce specifications without using a full blown proof assistant. In the late 90s while working at MVACS on the Mars Polar Lander, I found myself using the secure OpenBSD operating system to deploy the self-hosted service that @nickludlam and I have run together ever since. I became an OpenBSD developer with...

Started PhD at Cambridge

2002-09-01T01:00:00Z

I started my PhD at the Systems Research Group in Cambridge this week, based in the Computer Laboratory and Robinson College . I'll still be working part-time at NetApp , but my primary focus will be on the Xen hypervisor and other systems research topics.

Streamlining PHP on OpenBSD

2002-06-24T01:00:00Z

I've committed a big improvement to the PHP port on OpenBSD, by switching from a complex set of FLAVOR tags over to a set of independently installing "multi packages". The first thing I did was to import the core PHP without any extensions. commit 15dc0f67ef5fd0cae9fb841e608b90b9f51c71ca Author: avsm Date: Mon Jun 24 19:23:41 2002 +0000 Import php4-core-4.2.1 Installs the barebones php4 with only the gettext, iconv and recode modules compiled in. All of the other modules have to be installed as shared modules on top of this. In addition to the Apache module, this package also includes a php command-line binary which can be used in shell scripts. The binary uses the same /var/www/conf/php.ini file as the Apache module. There is some non-i386 breakage at the moment...

Streaming U2 live across the Internet

2002-04-01T01:00:00Z

After the Mars Polar Lander crashed , I took a job at NetApp working as the product architect for NetCache . Among the hundreds of deployments that I help setup across the world, the most fun was figuring out how to scale one of the biggest bands in the world at the time wanting to stream their concert live to a global audience. [… 158 words ] Anil Madhavapeddy and Alberto Crivelli . Technical report (TR-3152) at NetApp . URL (tr.netapp.link) BIB PDF

Xen Hypervisor

2002-01-01T00:00:00Z

Xen Hypervisor I was on the original team at Cambridge that built the Xen hypervisor in 2002 -- the first open-source "type-1" hypervisor that ushered in the age of cloud computing and virtual machines. Xen emerged from the Xenoservers project at the CL SRG, where I started my PhD and hacked on the emerging codebase and subsequently worked on the development of the commercial distribution of XenServer. Back at the turn of the century, the Computer Lab SRG faculty at the time (led by my first PhD supervisor Ian Pratt ) decided to start the XenoServers project, which would build a public infrastructure for wide-area distributed computing. An EPSRC grant lead to a number of graduate students all surging into the SRG in around 2002 to work on the project, including me. The later history of Xen...

Added a caching subsystem to Horde

2001-08-20T01:00:00Z

While hacking on making Chora performant enough to work as the official PHP CVS web-viewer, I added in a general caching subsystem into the Horde PHP framework. Do let me know if you end up finding a use for it in your own applications. Add in a Cache framework for persistent storage and retrieval of cached objects. Consider it experimental for now. Basically works for Chora's needs ... implements a filesystem driver which tries to act sensibly (writes to a tmp file, then does an atomic rename to the cache object), to avoid synchronization issues. It does not cleanup the cached repository at the moment - needs to have a garbage collection function done at some point. -- Anil Madhavapeddy

Chora now the production CVS viewer for PHP

2001-08-05T01:00:00Z

I spent a chunk of time through the year working on the Horde project. I began when I got commit to IMP webmail to fix some bugs in the MIME rendering for our Recoil deployment. You can see my code commits on the horde-cvs mailing list archive. After getting to grips with the PHP code, I then went on to totally rewrite the Chora version control viewer so that the CVS repositories for Horde could be browsed online instead of only via the command line. I'm extremely proud to report that the PHP project has now deployed Chora for production use to serve up cvs.php.net , making it our biggest user by far. Thanks for making my day, Rasmus! I switched Chora over to be the default web cvs system behind cvs.php.net now. The old viewcvs site is still available at viewcvs.php.net (dns may not have...

Added internationalisation to the Chora viewer

2001-08-03T01:00:00Z

One of the coolest things about hacking on the Horde framework is that it gives me lots of features for free that I can use in my web applications. The latest thing I added to the Chora CVS viewer today is the internationalisation framework, so that the frontend can be translated to multiple languages. I've added in a simple German translation to start with, but please contribute your own strings if you get the opportunity.

GPL release of OpenFX

2001-02-10T00:00:00Z

Slashdot covers the GPL release of OpenFX , which I worked on with Stuart Ferguson (my brother's PhD supervisor in Queen's University Belfast). It has a renderer and raytrace engine, NURBS support, kinematics-based animation, morphing, a plugin API - and it's under the GPL. Currently only for Windows, but they're working on a Linux and FreeBSD port.

I am now a core PHP developer

2001-01-09T00:00:00Z

I've been maintaining PHP on OpenBSD for a while now, including the core package distributed as binary packages. So as of today, the core team has decided I'm trustworthy enough to have my own commit bit to the central PHP repository, where I can commit code fixes and also maintain the OpenBSD on PHP official instructions. You can contact me on avsm@php.net if you need any help!

I'm now an OpenBSD developer

2000-12-26T00:00:00Z

I've been using OpenBSD for a few years now as the primary OS for recoil and have been contributing fixes and ports when I get a chance. So I'm incredibly excited to report that the project leader, Theo de Raadt, has invited me to become an OpenBSD developer. I've registered my keys now, and will be known as avsm@openbsd.org ! My first commit is to start fixing up the PHP port, which I have been working on in PHP-land for a while now. commit 93d5cc5ae56b22b19aa3bce34d38fa260b882d16 Author: avsm Date: Tue Dec 26 23:35:43 2000 +0000 - update to php-4.0.4 - bump NEED_VERSION - no longer need extra distfile number4.tar.gz since it has been integrated into the main distribution - ltconfig, mysql socket patches are in main distribution now, so they are removed. Note...

I'm now a Horde core team member

2000-10-16T01:00:00Z

After contributing some patches, I've now got the honour of becoming a core team member of the Horde project. Many thanks to Chuck Hagenbuch and Jon Parise for their trust in me! I'm planning on fixing bugs in IMP and the webmail subsystem, and am getting interested in version control and CVS as well, so I'm going to look at Whups and Chora ore. You can follow my commits on the Horde-CVS commit archives, where I am avsm@horde.org !

Paper on the NASA Mars Polar Lander website architecture

2000-07-01T01:00:00Z

Although the Mars Polar Lander ended up crashing , the website itself was one of the busiest websites in the world at the time during the approach to the landing. I was the person handling the website architecture and the amazing webmaster@mars.nasa.gov account at the time. I worked closely with Sun and NetApp and wrote up a technical report on how the Mars Polar Lander website acceleration architecture worked. Niall Doherty and Anil Madhavapeddy . Technical report (TR-3071) at NetApp . URL (tr.netapp.link) BIB PDF

Slashdot covers the Mars Polar Lander

1999-12-03T00:00:00Z

In my capacity as webmaster of the Mars Polar Lander, I submitted a note to Slashdot. Although our amazing distributed website took quite a beating (with some estimated 1 in 4 Internet users trying to access it simultaneously), the Lander itself sadly crashed . On the bright side, I got mentioned in a Sun press release because of the Sun Netra T1 servers they gave us to host the website! You can read more about the architecture behind the site in " Application of a Distributed Web Site Acceleration: Mars Polar Lander ".