Adding a device to btrfs RAID1
Published:
Adding a new device to a expand space on nearly-full btrfs filesystem in RAID1 configuration is easy. Rebalancing the filesystem to make use of the extra space is less so. It may appear that btrfs is almost completely ignoring the new device. Why?
As an example, let's take a btrfs filesystem in RAID1 configuration using two physical hard drives. For the sake of simplicity, let's assume that both of those drives offer 1 TB of space. The new drive that will be added offers 2 TB, and is supposed to double the available space.
The steps you take to add a new device to a btrfs filesystem are simple:
btrfs device add /dev/<new-drive> <fs>
Execute the above command and you are done. However, if you look at the filesystem or device usage statistics (they are the same data, just presented in another way) you will notice that the new device is not used:
btrfs filesystem usage <fs>
This is no good, because with RAID1 btrfs needs to be able to mirror data on at least two drives, and if there is only one drive with plenty of space and the others are almost full, the new drive will not be used. So you decide to rebalance the filesystem:
btrfs balance start --full-balance <fs>
This should be a costly operation, but it finishes in a few minutes. You inspect usage statistics again, and see that hardly anything changed. The new drive is still unused, and the filesystem still reports only a tiny bit of usable free space despite having plenty of unallocated storage.
To make sense of this problem you have to be aware of a couple of things about this particular configuration:
- btrfs allocates in chunks
- btrfs' RAID1 requires mirroring a chunk on two different devices
- balancing the filesystem requires allocating new chunks
Usually this works fine, but if all devices making up the filesystem were nearly full when a new device is added, a degenerate case rears its ugly head. The new device may be empty, but it does not matter if a chunk cannot be allocated on any other device. To visualise:
| drive | space (bytes) | allocatable chunks |
|---|---|---|
| A | 1 TB | 1 |
| B | 1 TB | 1 |
| C | 2 TB | 2137 |
A rebalance may start, allocate a new chunk on A and C, move some data to drive C, then do the same with B and C, and then finish, because it will be unable to do anything more. Why is it unable to do anything more? Because it has exhausted allocatable chunks on drives A and B, and therefore cannot allocate a mirrored chunk on two devices. When balancing a RAID1 btrfs you need available space on at least two devices.
The easiest way to solve this problem is to add another device for the duration of the rebalance. Having two empty devices the filesystem will be able to satisfy the constraint it was given. You may let a full rebalance run, or watch its status with
btrfs balance status <fs>
while also keeping an eye on usage statistics. Once you have enough unallocated space on the old devices you may cancel the balance, remove the temporary devices, and run the balance again.
The question that immediately appears is this: how much unallocated space is "enough"? There probably exists a proper answer to this question, but the rule of thumb I use is 10-20% of the old filesystem (depending on what I have available at the moment). In any case, a couple dozen gigabytes ought to be enough for anybody.
As soon as you see that the sum of unallocated space on the old devices is bigger than the allocated space on the temporary device you can cancel the first rebalance, remove the temporary device, and start a new rebalance (only involving the target drives).
Previous: Working 1 Gbit/s Ethernet USB-C adapter