Storage Spaces and the lack of built-in data balancing functionality

I blogged about changing my home media setup late last year on my HP Microserver N54L after migrating from Windows Home Server 2011 to Server 2012 R2 Essentials and choosing to use the Storage Spaces feature with a parity configuration. One key area that became apparent more recently is the lack of data balancing in any given pool configuration. This is potential trouble spot for a few reasons. I’ll go through the pains and workarounds of getting data balanced on a Storage Space pool.

There is no data balancing functionality in Storage Spaces

So I’ve already stated to mention, no data balancing feature exists in Storage Spaces currently, neither automatic or through PowerShell. This includes Windows 8/8.1 and Server 2012/R2.

My discovery of the lack of data balance became quite apparent when my original parity pool of 4x 2 TB Western Digital Green Drives were all over the 70% usage mark (default warning value), queue this scenario here:

To add a bit of background to my setup, the four 2 TB drives were part of the pool at the very beginning, the additional 3 TB drive was added very recently after getting the capacity warning in the image above. The new 3 TB drive has been active in the pool for about a week, during this period I would estimate that about 60 GB of data has since been written to the pool since its introduction to the system. I can report that the usage of the new 3 TB drive is 2.49% at this time of writing, the usage ratios are certainly not proportionate to all other drives. To further compound the problem, the 2 TB drives that are filling up are still being used quite extensively, they obviously hold the majority of the pool, so their capacity is decreasing still quite rapidly, despite all the space on the 3 TB drive being available.

The warning to add two additional drives seems quite confusing at first, as the pool has plenty of capacity in terms of raw space, but Storage Spaces isn’t bothered about that. What its actually looking at is each drive individually and noticing that 4 of them in the pool are becoming full. What its actually seeing or rather pre-empting is the potential problem of 3 drives being full and no longer able to provide the resiliency required for parity. Hence the warning to add more drives. Compounded by the fact I have literally have no more SATA ports left, I can’t simply add more drives, like Storage Spaces wants.

Changing the warning threshold

As mentioned previously the default warning value is 70%. You can actually modify this to something higher if you wanted:

Set-StoragePool -FriendlyName "Friendly Pool Name" -ThinProvisioningAlertThresholds 85

This however won’t solve anything other than stop the warning from showing up until the new defined threshold percentage is hit. 70% does seem a little too cautious however, so I did adjust it to the above.

Forcing data balancing

So now the scenario and problems are clear, what are the options to fix it? Here are the potential options I discovered that can be used:

• Add more drives (Not really possible in my case)
• Moving data off and back onto the pool
• Upgrade the 2 TB drives with higher capacity
• Force a repair/rebuild by pulling and reintroducing the 2 TB drives

Adding more drives to the pool

While obviously the easiest solution, I’m all out of SATA ports on my HP Microserver N54L, so adding more drives like Storage Spaces wants isn’t as simple as is sounds. In reality, the HP Microserver Gen7 models technically only has 4 SATA ports for hard drives, but with some hacking (hacked BIOS and some mods) you can get 6x Internal Drives (Run one drive off the DVD SATA port) and run the OS on an SSD (ESATA – SATA) and just mount it with some industrial Velcro insde the case somewhere. This is without adding an additional RAID card of course, which would give you more SATA ports, but decent RAID cards cost a fair bit of money!

What about USB drives you might ask? In theory I could use the USB 3.0 PCI-E card I added a while back and introduce USB 3.0 drives to the pool, however, I’d like to keep a couple of these ports available for rotating server backup drives/general ad-hoc use of external hard drives. I certainly wouldn’t want to use USB 2.0 in the pool at any point.

Moving data off and back again

To try and even out usage, I could move data off the pool and then back onto the pool again. For this I would need a temporary space (which could be another pool) or just an external drive with enough capacity to hold data for a short time. The idea is, moving data off the pool will bring down the usage over the drives currently in use (mainly the 2 TB drives) then when the data is reintroduced back onto the pool, it is spread out again, this time the 3 TB will be used more. This however means moving my data around in a potentially less protected state without the same resiliency and will take a fair bit of time as I have terabytes worth of data on the pool. It is however, a fairly easily solution to execute. I also do have some redundancy as I do daily Server Backups to external USB 3.0 drives, which would be unaffected through the entire process. Though I may want to suspend backup while the pool is being messed with.

Replacing the 2 TB drives with greater capacity

So the lack of SATA ports means I can’t add more drives conventionally, but what about replacing the 2 TB drives in use now with greater capacity i.e. 3 TB or even 4 TB drives? This is possible because of the parity and resiliency, in theory I could literally pull one of the 2 TB drives right now and the pool would still work. It would be degraded, because its lost the redundancy, but if I immediately swapped in a higher capacity drive and added it to the pool, it could then repair and rebuild itself with the new drive in play. To do this I would have to replace each 2 TB one at a time and wait for the repair/rebuild operation to complete before repeating the process, as doing multiple drives in one go would kill the pool entirely. The downside to this is the repair/rebuild time, which is pretty lengthy, especially with parity. It is also an expensive solution replacing 4 drives in one go, but could be done incrementally overtime and potentially resolve the issue.

Reintroduce the 2 TB drives back to the pool

Another option I thought about was essentially creating a fake hard drive failure scenario, where by one of the 2 TB drives is pulled formatted and then introduced to the pool again, the pool would see it is a new drive and once this happens a repair/rebuild process will occur on the pool. This however, while a quicker solution would eventually rear its head again later, because the capacity of the drive is still the same, the usage would probably end up getting to the same place its at now sooner rather than later, but will make use of the 3 TB drive more

What’s the best solution for manual data balancing?

Simple answer, there isn’t one. Based on everything explained above I’d recommend these options depending on the situation:

• Move data off and back onto the pool (short term)
• Formatting one of the most used drives and reintroduce it back into the pool (short/medium term)
• Upgrade 2 TB drives with higher capacity (long term)

Its important to highlight that you need to have some resiliency when performing any method requiring you to remove or replace drives.

Changes in Windows Server 2016

Windows Server 2016 contains new and tweaked features for Storage Spaces, one area that’s caught my attention after reading this blog post is the addition of the Optimize-StoragePool command that has been added which does do data balancing.

Optimize-StoragePool -FriendlyName "Storage pool name"

Currently it seems this functionality works on mirror configurations, it doesn’t appear to be working on parity spaces. I wouldn’t read too much into that currently however, as Server 2016 is still in the technical preview stage, which is technically not even BETA yet. I assume by the time Server 2016 goes RTM this functionality will be more polished and likely be automated and run during the optimisation window as you’d expect