Backing up your LiteFS cluster

While LiteFS replicates data between nodes and supports failover, it does not currently have a disaster recovery system in place in the event you lose your entire cluster. These are some strategies you can use to reduce your risk of data loss in the event of a bug or disk corruption.

Periodic backup via export

The simplest approach is to export your database via the litefs export command and then upload that file to long-term storage such as S3. The export command is safe to use on a live database and can even be run remotely by specifying the -url flag.

Once the database is downloaded, it can be compressed & uploaded to S3.

# Download current snapshot the "my.db" database to a local file.
litefs export -name my.db /path/to/backup

# Compress the file.
gzip /path/to/backup

# Upload the file to S3.
aws s3 cp /path/to/backup.gz s3://mybucket/backup.gz

It’s also recommended to perform a rolling backup based on either hour, day or month depending on cost requirements.

# 1-day, rolling hourly backup
aws s3 cp /path/to/backup.gz s3://mybucket/backup-`date +%H`.gz

# 1-month, rolling daily backup
aws s3 cp /path/to/backup.gz s3://mybucket/backup-`date +%d`.gz

# 1-month, rolling hourly backup
aws s3 cp /path/to/backup.gz s3://mybucket/backup-`date +%d%H`.gz

This backup can be run on any of your nodes as even the replicas should have minimal lag behind the primary. If you run this on multiple nodes (such as all your candidates), make sure they are backing up to different storage locations so they do not overwrite one another.

Continuous backup via Litestream

Litestream is a tool for continuously streaming incremental backups to S3-compatible object storage. You can use it on top of LiteFS to provide point-in-time recovery in case your cluster fails for any reason.

This approach only works if you are running a static lease on your cluster.

Initialize your bucket

First, you’ll need to create a bucket on S3-compatible storage. In this example, we’ll use Tigris since it integrates nicely with Fly.io.

fly storage create

This command will return several configuration variables. Set the AWS access key ID and secret access key as secrets in your application:

fly secrets set AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...

Set up your configuration file

Once your bucket is created, you’ll need to create a Litestream configuration file. In this example, change my.db and mybucket to your respective database file and bucket names:

dbs:
  - path: /litefs/my.db
    meta-path: /var/lib/litefs/my.db-litestream
    replicas:
      - type: s3
        bucket: "mybucket"
        path: "my.db"
        endpoint: "fly.storage.tigris.dev"
        force-path-style: true

If you save this file to /etc/litestream.yml then Litestream will pick it up automatically for any Litestream commands you use.

The meta-path should be a path on a persistent volume. In this example, we’re assuming that /var/lib/litefs has been mounted from a volume.

Running Litestream

Litestream should be run after LiteFS has been initialized so we can specify it in the LiteFS exec. Litestream can run its own exec so you should specify your application there.

# litefs.yml

exec: "litestream replicate -exec myapp"

Litestream should only be run on the primary node. It requires write access to the database so it will not run on read-only replicas.

Verifying your setup

Once you have your application running, you should be able to run Litestream’s restore command to fetch the current version of the database:

litestream restore -o ~/myrestored.db /litefs/my.db

If you need to recover from a loss of the database, you’ll need to restore your SQLite database to a file and then use litefs import to import it into your cluster. Litestream cannot restore directly into your LiteFS mount.