Garage, Seven Months Later

PUBLISHED
16 NOV 2026
FILED UNDER
  • garage
  • self-hosting
  • object-storage
  • zfs
  • storm-buckets

Garage does one thing, and it does it well.

In April I set up a single Garage node on a Canadian VPS and synced my Obsidian notes to it. Since then it has become the storage engine under Storm Buckets, and I've been hosting it for many people at once. I still really like it.

I've also learned a lot about it, mostly the slow way. I haven't actually tried Garage geo-distributed yet. That's the canonical way to run it, the one with the most research and proof behind it, so I'm not worried about it. What I'm doing right now, ZFS on one machine, is the experimental way to run Garage, and it's the one that lacks documentation. Most of what's below I learned through practical use.

Garage is lightweight

Garage stores objects and speaks S3, and it stops there. So you have to prop it up and build other systems around it, and what you build depends on where you're putting it. I'm putting it on the public internet, which decided most of mine.

The web server

The most obvious one, the one everyone starts with, is the web server. Garage has no TLS of its own, so something has to stand in front of it, and in my case that's Caddy.

Caddy ended up doing more than certificates. Storm Buckets has a file browser that signs requests from the customer's own browser, so the proxy has to answer the CORS preflights.

The proxy can also hurt you. My standard security headers sent X-Frame-Options: DENY on every response, and a customer's app couldn't show a PDF from their own bucket in an iframe. We had both been looking at CORS, and I had never tried that program myself.

s3.example.ca {
	reverse_proxy 127.0.0.1:3900

	header {
		defer
		X-Content-Type-Options "nosniff"
		Referrer-Policy "strict-origin-when-cross-origin"
		# no X-Frame-Options: DENY stops a customer's app
		# showing their own object in an iframe
		-X-Frame-Options
	}
}

I would never vhost

I serve S3 path-style only, and I would never do it the other way, personally.

# path-style: takes no effort, and you lose nothing
https://s3.example.ca/photos/cat.jpg

# vhost-style: takes a lot of effort, and you lose features
https://photos.s3.example.ca/cat.jpg

Putting the bucket name in the hostname is a cosmetic-only feature that needs a wildcard certificate and a DNS challenge my Caddy build doesn't ship with. So it's a lot of effort for no return, and you're actually losing out on a feature, local aliases.

I also leave root_domain unset under [s3_api], because if the endpoint's own hostname sits under that suffix, Garage reads the endpoint's label as a bucket name and every signed call comes back 404.

The orchestration layer

I'm selling storage to people who sign up on a website, so something has to create their buckets and keys on the node. I built a custom orchestration layer for that, Storm Pulse.

Since the node is on the public internet, I need the admin calls locked tight and never on an open port, so it only ever connects out to my Django dashboard over a WebSocket.

The Guard

The most important one is the Guard. It's a custom Pingora proxy that sits between Caddy and Garage, and its whole job is to protect your account and the node it runs on. It's the only closed layer I run on a node.

Garage is flexible

There are so many ways to run Garage, and that means there are a lot of footguns.

The biggest one is redundancy. There are two ways to get it. Local redundancy is ZFS or hardware RAID, where one machine survives its own drives failing. Geo redundancy is the way Garage was built for, where machines in different places each hold a copy. You really never need both.

Geo redundancy

I haven't run two copies, and from Garage's configuration reference I wouldn't. With replication_factor = 2, one node down leaves your data readable and makes every write fail.

Three is the sweet spot. One node goes down and you still have full service, and the time to fix the one. Two go down and you've still lost nothing. You can't write, and reading takes switching Garage to its degraded consistency mode, which is of course better than losing data, and gives you even more time to fix the problem. The geo-distributed plan on the Storm roadmap is built on that number, three cities holding the same block.

Local redundancy

I run the local kind. Each of my endpoints is a single node, so the redundancy lives in the disks under it, and my weekly checkup is about the one copy that exists. A failed drive is covered, and the rest of what you risk with one node is written down on the roadmap.

Garage's deployment guide tells you to skip ZFS for the data directory, since Garage checksums its own blocks and XFS is faster. That's advice for the geo kind, where ZFS redundancy underneath would pay for the same safety twice. On a single node the disks are the only redundancy there is, so I put the data on raidz3.

That put me somewhere with almost nothing written about it, and the settings I ended up with came from benchmarks I ran myself:

garage-meta   recordsize=16K   compression=lz4   atime=off
garage-data   recordsize=1M    compression=zle   atime=off   xattr=sa

The data record size matches Garage's 1 MiB block size. The one that surprised me is compression. I first turned it off for the data, because Garage already compresses blocks with zstd and compressing twice is waste. Then I measured 50 MB objects taking about 100 MB on disk. Garage keeps the zstd output even when it's no smaller than the input, and adds a small header, so a block of random data comes out a few bytes over 1 MiB and takes two 1M records. With zle, which only squeezes runs of zeros, the same objects took about 51 MB.

The metadata database

This one is super important. Garage keeps its index of every object in an LMDB database, and Garage's own reference says LMDB is prone to corruption after an unclean shutdown. ZFS can't fix that for you. A checksum proves a file holds the bytes that were written, and it can't repair a database that was written half-finished. So two settings stay on, ZFS or no ZFS:

db_engine = "lmdb"
metadata_fsync = true
metadata_auto_snapshot_interval = "6h"

metadata_fsync is off by default. The snapshot interval gives me a copy of the index that is at most six hours old, and my weekly checkup fails if the newest one is more than a day old.

The Garage team is working on a new metadata engine that makes a crash safe at the source.

Fsync has a price

Syncing every metadata write is paid for on every upload. In my benchmark, four times the cores bought twice the uploads per second with fsync off, and only 1.35 times with it on. Metadata was sharing the raidz3 pool with the data, so every PUT waited on small synced writes to a parity pool. The best advice is to put the metadata on an SSD.

Local aliases

Local aliases are an amazing feature. A bucket name belongs to the key that made it, so nobody has to fight over bucket names. But you have to understand that you're opting into a lot more effort and a lot more care.

Deleting is one example. S3 addresses a bucket by name, and with local aliases the name belongs to the customer's key, so my own tooling can't reach their bucket by name the way it could with a global one. It has to go by the bucket's id through the admin API, or give a temporary key an alias of its own first.

CORS is the other. Almost nobody will worry about CORS. But I'm running this multi-tenant and charging money for it, and my clients want multiple layers of defense, so they care. A preflight request isn't signed, which means there's no key on it, and without a key Garage can't tell which bucket a local alias points to. You can't do it on preflight, so you need your own custom handler. Mine enforces each bucket's rules on the signed request, in the Guard. I sent a fix upstream for part of this.

Bringing up a real node

The first dedicated server taught me the hardware has footguns of its own, before Garage even starts.

The drives arrived as volumes behind a RAID controller, where ZFS wants the raw disks. Sales had told me pass-through "can be done", and I should have asked for the controller mode in writing, in the words the vendor's own menu uses. The default install had put the operating system on one of the twelve data drives. smartctl couldn't read drive health until the controller was converted.

The one I'm gladdest I caught was in a rehearsal. I built the node's disk layout in a virtual machine with twelve disks and pulled the first one. The firmware booted from another drive and grub loaded the kernel, and then the initramfs waited with no limit for a partition that lived on the missing drive. Finding that on my laptop cost an afternoon. Finding it on the node would have meant the first dead drive takes the whole box down.

What I haven't done yet

In the first post I wrote that the path to a multi-node Canadian cluster was built into Garage's design, and that it was what excited me. Seven months later I haven't tried it. Today every endpoint is one node, and the plan to hold one block in three cities is on the roadmap, with no date on it. I'm bootstrapped, so each node waits until I can pay for it and set it up properly. It's still what excites me, and it's the well-proven half of Garage, so the experimental half is the one I'm running first.

Giving back

Running Garage this hard is also how I found things to fix in it. The fixes I've had merged are listed on the Storm roadmap, and I wrote about the first two.

Would I pick it again

I would still pick Garage, of course I would! The only downside is that it's not feature complete (no object lock or versioning), but I love how flexible and lightweight it is, and the freedom it gave me to make a much more robust system around it.

Storm Buckets is S3-compatible object storage hosted in Canada, built on Garage.

Have comments? Add me on Mastodon and shoot me a message.