Drive Balancing tool for WHS

[Beefcake550]

Hi All,

I've finally gotten time to finish up this app I've been working on. It basically balances all the data on your drive pool equally. Let me know if you like it! I thought it could be helpful with Sage setups since I'd think balancing can get all out of whack with large amounts of recordings.

Here's the readme:

UPDATED NOTE:
Some people have been having trouble not getting any data to move when their D:\ has lots of data on it. Before doing anything else, please run Drive Balancer to clear the landing zone only. To do this, choose "n" to include d:, then choose "y" to clear landing zone, and the "5" to exit. After this initial run (which can take hours), rerun Drive Balancer the with the options you want. I believe this will help the problem of no data being moved on some people's systems.

Usage Notes:
This program was designed to "even out" the data on a drive pool for WHS. The main purpose was when a person adds a drive to the pool, it doesn't necessarily start getting used for a while. While Microsoft has made changes to the usage of the Landing Zone (as in it doesn't exist anymore), some people's systems might still be in a non "evened out" state as wish to correct this. On my own system, I routinely get within 2% of perfect balancing on all pooled drives.

For a walk through of the tool running, checkout the walk_through.txt file.
Testing:
I have tested this program on my own system which consists of:

System Disk:
Areca 1210 RAID5 set consisting of (4) 250GB Seagate Drives = 20GB C: and 678GB D:

Pooled Disks:
(3) WD 1TB Green HDs
(2) Seagate 1.5TB HDs


IMPORTANT NOTE:
With some help from user "S_M_E" on SageTV forums, I am able to "restart" the Drive Extender service which is responsible for doing the actual balancing. The program detects a restart is needed if <5MB/s of total disk throughput is monitored on all pooled drives. Monitoring disk activity would not have been possible within Sam Wood's help. At this point, it restarts the service and marches along. Feel free to restart the service on your own if you are watching the process and see no disk activity (using Disk Management plug-in).

You WILL see some critical WHS warnings reported all related to storage space. Do not worry, this is expected behavior while the tool is running and will be cleared up at the end of the run.

If you need to hit Ctrl-C out of the program even though it is automatic and will end on it own. If you feel the program is behaving badly, and you Ctrl-C out of it, You will have to manually delete the following DIRs:

C:\fs\*\fake\
D:\fake\

I have extensively tested this program on my own system with my own data. I have lost nothing and you should realize that this program does not delete any data (It only creates fake files). As such, I can't guarantee that any data on your own system. Don't be afraid, either do a backup first, or jsut trust me and run the tool anyway!

Acknowledgements:
Theo van Elsberg - author of LZReallocator
Brent from wegotserved WHS forums
S_M_E from SageTV forums
Sam Wood - author of Disk Management add-in

======================================================================================

Revision History:
v1.05
- Attempting to solve problem where no files are moved on the system when D:\is filled
prior to running Drive Balancer. Please read UPDATE NOTE for potential solution
- Fixed bug where calculation of free space on D: was not "worked around". Refer to v1.03
history notes below. This fix will help systems that either
a) don't include D:\ in balancing
b) tried to clear the landing zone
- Added programmable time interval for checking disk activity
- Added programmable disk activity threshold for restarting Drive Extender service

v1.04
- Fixed bug where D: was not always cleaned up
- Modified restarting of DE service to monitor disk activity. This will restart DE within
seconds instead of 15 minutes. Should improve run time on systems with large files.

v1.03
- Handled bug where WHS returns free space of D: as (C: + D
- Fine tuned "crunching" algorithm so balancing should be better on different sized HDDs
- Added exception handling around creation of fake directories and fake files

v1.02
- Added option to allow D: to be part of balancing.
- Fixed bug where restarting DE would not occur and forward progress would likely not occur.

v1.01
- Fixed bug with option #2 and a drive was large enough to contain an equal amount of data.
- Added option for entering in your own data
- Added "quit" option in case all you wanted to do was clear the landing zone

[S_M_E]

Can I say yes to 1 and not pick 2 or 3?

[Beefcake550]

You can say yes to #1, and then choose either 2 or 3. It will make a few calculations and then ask if it seems reasonable. You can then hit 'n' and it will clean up D: and exit.

-Brian

[S_M_E]

Does #3 try to equalize used space or free space? With different sized drives that could be a big problem if it's used space.

[Beefcake550]

#3 equalizes used data on each drive. So, say you have 2TB total data used on the pool and 5 200GB drives, there would be (2TB/5 drives = 400GB) on each drive.

It would be easy enough to add a equalize free space option. I could whip that up quickly...

-Brian

[S_M_E]

What if you have 2 250G drives, 2 500G drives, 2 1T drives and 2 1.5T drives with 1T free?

EDIT:
Doing the math it seems it would try to put ~687G on each drive which is impossible for the 250G and 500G drives. How would your tool handle that?

With free space equalization it would make 125G free on each drive which is wildly different percentages but at least the tool isn't trying to put 687G on a 250G drive.

[Beefcake550]

well, 1TB free / 8 drives = 250GB free each disk. So, it would seem like your 250G drives would be empty! I'm sure that's not what you want.
What are you looking for?

[Beefcake550]

I guess I could put in an option where the user can put in the used/free space for each drive as they wish, but that can get very problematic to find error conditions with.

Using the equalize percentage used on your situation above, you'd get:

Total space = 6500GB

Free space after balancing:
250GB HDs = 38.46GB
500GB HDs = 76.92GB
1T HDs = 153.86GB
1.5T HDs = 230.7GB
So, each disk would have 15.38% of it's capacity free.

-Brian

[Beefcake550]

Update to equalize the free space, but no checks to see if we're allocating too much free space to a drive, so be careful. I'll implement that check later.

-Brian

[Beefcake550]

There is a check to make sure that too much data isn't getting put on a drive.

[S_M_E]

Originally Posted by Beefcake550

well, 1TB free / 8 drives = 250GB free each disk.

125G

Originally Posted by Beefcake550

I guess I could put in an option where the user can put in the used/free space for each drive as they wish, but that can get very problematic to find error conditions with.

Using the equalize percentage used on your situation above, you'd get:

Total space = 6500GB

Free space after balancing:
250GB HDs = 38.46GB
500GB HDs = 76.92GB
1T HDs = 153.86GB
1.5T HDs = 230.7GB
So, each disk would have 15.38% of it's capacity free.

-Brian

Yeah, equalizing by percentage or free space wouldn't seem to cause problems but what does your tool do if you try to put 687G on a 250G drive?

Originally Posted by Beefcake550

There is a check to make sure that too much data isn't getting put on a drive.

So, using the used space equalization method, the 250G, 500G and 1T get full -10G and the 1.5T drives have ~470G free each?

[Beefcake550]

yeah.. hehe 125GB.

I'm gettign confused by what you are asking. It is safe to run the tool to the point where it will ask you if the calculations it made seem right. I've put in range checking except for that new option of equalizing free space. You can exit the program by answering "n" when it asks if "it seems reasonable".

-Brian

[S_M_E]

The first time I ran the first version I entered Y for D and after D was done I entered 2 and it crashed. You may want to have it create a log file. Now I'm using #1...

[Beefcake550]

WeGotServed forums found the same bug. Sorry about that. I've since fixed it and added a few more options with checking that should make you happy.

-Brian

[S_M_E]

When using the % option I had a wide range when finished, from 78% to 92%. It's still more balanced than it was though and it cleared off D well.

[Beefcake550]

Givent he smaller sized hard drives in your system, it doesn't surprise me. I built in some padding such that it wouldn't be too aggressive. Each should end up within about 20-25G of each other Let me know if that's not the case.

-B

[S_M_E]

Originally Posted by Beefcake550

Givent he smaller sized hard drives in your system, it doesn't surprise me. I built in some padding such that it wouldn't be too aggressive. Each should end up within about 20-25G of each other Let me know if that's not the case.

-B

The drives I used in that other example aren't exactly the drives I have in WHS but just comparing 3 500G drives out of all my drives the range is 78% to 90% which is about a 57G difference in free space. *shrug*

[crarbo1]

I think that I will end up using this utility but I do have a question. I use my WHS as my SageTV server as you probably expect but I was wondering if I could use this tool while SageTV was recording to the pool? If not, then I could wait until no one is recording anything, watching a recording, or watching live TV to do it. Just don't want to screw anything up.

Also, I assume all I have to do is unzip the files to a directory in the WHS and run from that directory. Is that correct?

Thanks,
Chuck

[Beefcake550]

That's a tough one crarbo1. I would think it be safe, but I'd feel much safer if you just stopped the SageTV service in order to prevent any recordings or disk activity besides this tool.

-Brian

[crarbo1]

Originally Posted by Beefcake550

That's a tough one crarbo1. I would think it be safe, but I'd feel much safer if you just stopped the SageTV service in order to prevent any recordings or disk activity besides this tool.

-Brian

I agree but I thought I would ask anyway. I will just have to do it when I have a few hours that it is not in use. That may be hard though.

Thanks,
Chuck