A tool for backing up ATProto related data to S3
willdot.net
1## Back AT it
2
3This is a tool I'm activly developing to back up my ATProtocol type things to S3 storage.
4
5Currently there are 2 things that can be backed up:
6
71: PDS data - A users repo and their blobs
82: Tangled Knot data - The repositories directory that contains all of the repo data and the directory that contains the SQLite database
9
10The PDS repo data is pulled straight from the xrpc endpoint at sent straight to S3. The blob data however is streamed into a zip file and sent to S3 so that not all the data is held in memory while the backup takes place (the minio library will still keep some in memory as a multipart request).
11
12It's very hacky right now and needs polishing to use with caution. Although let's face it, the worst it can do at the moment it backup some bad data which is better than no data 🤪
13
14
15### How to use
16
17Clone the repo and copy the `.env.example` file to be `.env`. Fill in the `.env` file with you S3 variables.
18
19For PDS data backup you need to ensure that `DID` and `PDS_HOST` are populated. (You can run this tool on any machine to back PDS data up)
20
21For Knot data backup you need to ensure that `TANGLED_KNOT_DATABASE_DIRECTORY` and `TANGLED_KNOT_REPOSITORY_DIRECTORY` are populated. (You need to run this tool on your Knot server to back up Knot data)
22
23Run `go run .`
24
25### Todo
26
27- [ ] - Turn this into a long running app using a cron library perhaps
28- [ ] - User query params properly when creating the URLs to fetch repo and blobs
29- [ ] - Allow configuring the backup of knot repo data per users DID maybe?