How does SageTV software decide what drive to use?

[MrVining]

In a multi HDD set-up how does SageTV software pick the drive it will use to store a show?

[Opus4]

This is covered in Appendix C of the SageTV PDF manual.

- Andy

[evilpenguin]

Right now? Not so smart.

Sage just picks the HD with the most free space and then records to it. Obviously, the big drawback of this is that if you have one drive with a lot of free space Sage will *exclusively* record to that drive with *all your tuners* until it reaches equilibrium capacity with the other drives. Some people have no problem with this, I think it causes sluggishness, it really depends on a lot of factors. (Just this week I caught Sage recording 5 HD shows to the same drive at the same time while my other 3 recording drives sat idle)

Currently, the only thing you can do to change this is to force specific tuners to record to specific drives (forced_video_storage_path_prefix). But, this has its own nasty side effect where-in when one of your tuners fills up it's drive it'll start deleting shows even though you have plenty of free space on other drives.

I submitted a bug about a year ago about this, and they say they have a much more robust solution ready to go, but they didn't want to mess with it during the last 2 beta rounds. So hopefully we'll see something about it in the next one.

[FreshOne]

Originally Posted by evilpenguin

Right now? Not so smart.
they say they have a much more robust solution ready to go, but they didn't want to mess with it during the last 2 beta rounds. So hopefully we'll see something about it in the next one.

I recently upgraded my Sage setup to 2 HDHRs (4 tuners) and would welcome an easier and more intelligent way to split the recordings among multiple HDs.

[nick_l]

Eenie Meenie Miney Moe (sorry, couldnt resist)

[Fuzzy]

I would love more of a round robin system to drive selection, where it chooses the unused drive with the most free space. Then, if there isn't an unused drive, it chooses the single use drive with the most space. If still no go, it chooses the dual use drive with the most space, and so on. This should do a pretty good job of evening the load. Drive use should get a point for each recording in progress to it, and each playback. Perhaps something where HD gets 2 points, and SD gets 1, so you could even out the load a bit. Would certainly be a GREAT improvement over the current system.

I would also like to see a drive leveling system in place, where it would periodically analyse the drives, and look for recordings that can be moved to even out the free space among the drives. This would help in the drive selection system.

[briands]

What does WHS pooling do to the mix? If all recordings are going to the pool, I suspect Sage does nothing special since it appears to only have one drive available. What if you have a share in the pool and other drive(s) outside the pool?

[gplasky]

Originally Posted by briands

What does WHS pooling do to the mix? If all recordings are going to the pool, I suspect Sage does nothing special since it appears to only have one drive available. What if you have a share in the pool and other drive(s) outside the pool?

If Sage is configured for the pool drive it is only writing to one drive in WHS (the pool drive) and has nothing to calculate. If it is configured for a the pool drive and a drive out of the pool it would write to the drive with the most free space I would think. For a better understanding of how WHS works google for the Technical Brief Windows Home Server Drive Extender document. Here is a partial explanation how WHS handles it.

The Windows Home Server Drive Extender Filter determines which hard drive to initially write a file to. One goal of the algorithm the Filter uses is to keep related files together on the same hard drive. Copying music from a CD to a hard drive illustrates why this is important. If a single secondary hard drive failed, it is more convenient to lose all the music from a few CDs and then re-copy those CDs than to insert hundreds of CDs to re-create one track from each. One way to achieve this is to ensure that a set of files created around the same time are stored on the same secondary hard drive.

An obvious method for choosing the secondary hard drive would be to use the one with the most space free, but that would result in sometimes alternating among secondary hard drives. Consider the CD scenario again. A moment ago, the second hard drive had the most space available, so Track 1 of this CD was saved there. Now that the second hard drive has this new file on it, the third hard drive has the most room to hold Track 2, but we would really prefer to store it on the second hard drive with Track 1. Choosing the hard drive with the least available space is a good choice because the hard drive with the least free space tends to remain the hard drive with the least free space for a long time, and the same secondary hard drive will be chosen.

The Drive Extender Migrator service uses the secondary data partitions with the most free space to store the secondary or alternate shadows of new files. When Folder Duplication is enabled, the Migrator service fills the secondary data partitions with the most free space, and uses the primary data partition as a last resort.

Another goal in selecting the secondary hard drive for a file is to ensure that migrated files have room to grow. If a migrated file is later opened and data is added to it, you need enough free space on the secondary hard drive to hold that new data. This suggests that there should be a buffer of free space remaining on the lowest-space secondary hard drive before placing a shadow file on it. The Drive Extender Migrator service will attempt to keep a buffer of 10 GB on each hard drive and increase that buffer to 20 GB of free space, by migrating files to other secondary hard drives that have 20 GB or more free space available.

If all of the secondary hard drives are so full that there may not be room for existing shadow files to grow, the Migrator service may move some shadow files to the primary data partition or alert the user that the home server is running low on storage. The Migrator service makes the appropriate updates to the tombstones whenever it moves a shadow from one secondary data partition to another or from a secondary data partition to the primary data partition.

Gerry

[RobJ]

Originally Posted by Fuzzy

I would love more of a round robin system to drive selection, where it chooses the unused drive with the most free space. Then, if there isn't an unused drive, it chooses the single use drive with the most space. If still no go, it chooses the dual use drive with the most space, and so on. This should do a pretty good job of evening the load. Drive use should get a point for each recording in progress to it, and each playback. Perhaps something where HD gets 2 points, and SD gets 1, so you could even out the load a bit. Would certainly be a GREAT improvement over the current system.

I would also like to see a drive leveling system in place, where it would periodically analyse the drives, and look for recordings that can be moved to even out the free space among the drives. This would help in the drive selection system.

I agree, and would like to see what Sage has come up with, for load balancing.

For now, I do drive leveling manually. I make sure that 2 local drives always have the most space, and stay within 1GB of each other. I manually move recordings elsewhere, to my desired destination for that recording, and then move recordings between these 2 drives to balance them. Works great. I never have more than 2 recordings happening to the same drive at the same time.

[Fuzzy]

It's ridiculous that I have 5 sage storage drives, and 5 tuners, and still, on occasion, have 3 recordings on one drive...

[GKusnick]

Originally Posted by Fuzzy

I would love more of a round robin system to drive selection, where it chooses the unused drive with the most free space. Then, if there isn't an unused drive, it chooses the single use drive with the most space. If still no go, it chooses the dual use drive with the most space, and so on. This should do a pretty good job of evening the load. Drive use should get a point for each recording in progress to it, and each playback. Perhaps something where HD gets 2 points, and SD gets 1, so you could even out the load a bit. Would certainly be a GREAT improvement over the current system.

I think you need to be careful about making the rules too complicated. There's a wide variety of disk hardware out there, and what looks like a single drive to Sage could in fact be a RAID array or NAS device with its own internal load-leveling logic. Even single drives these days have internal command queuing logic to optimize throughput. Layering another set of rules on top of that without knowing what the device itself is doing could actually degrade performance in some cases. For instance I can imagine a case where writing two streams to the same fast internal disk gives better overall performance than offloading one of them to a lower-speed external drive or network device.

[evilpenguin]

Originally Posted by GKusnick

I think you need to be careful about making the rules too complicated. There's a wide variety of disk hardware out there, and what looks like a single drive to Sage could in fact be a RAID array or NAS device with its own internal load-leveling logic. Even single drives these days have internal command queuing logic to optimize throughput. Layering another set of rules on top of that without knowing what the device itself is doing could actually degrade performance in some cases. For instance I can imagine a case where writing two streams to the same fast internal disk gives better overall performance than offloading one of them to a lower-speed external drive or network device.

Yeah, but *some* new rules are needed cause the current implementation could end up witting two streams to the slower drive while the faster one sits idle.

[Fuzzy]

A system could easily be made that assigned 'points' not only to the stream, but to the drive itself. The local RAID could be assigned more 'points' and therefore could receive more than one HD stream at a time before rotating to the next.

[GKusnick]

I agree that the current scheme is less than ideal, and that it's not hard to come up with new rules that sound better on paper. The hard part is knowing how well the proposed rules will perform in real life on a variety of user hardware. Maybe the ultimate answer is for the Config Wizard to run performance benchmarks on each configured recording directory to try to measure how much throughput it can take under various conditions. That seems preferable to a scheme that requires users to configure drive priorities manually in order to get good performance.

[PAF]

With 64K clusters, is this really important? I remember having this discussion before and it ended up being that you need about 10 HD streams to top out the bandwidth of a 7200 RPM hard drive (or something along those lines)...

Has anyone actually tested this as an issue? Something like record 1 stream for 5 minutes... then 2 streams for 5 minutes... etc. until stuttering is seen or artifacts or any other weird stuff. I tried 3 streams to 1 hard drive and I haven't seen that much difference, has anyone tried this with other results?

Is there a way to document these kinds of anomalies?

[Fuzzy]

if everything was ideal, sure, a hdd could handle a lot, however, many have experienced the inconsistent results with as little as 3 recordings on a single drive. Maybe this could be fixed by some better buffering systems, but the better method would be to just fix the drive selection.

[GKusnick]

The problems arise not so much from multiple simultaneous recordings, as from recording to and playing back from the same disk at the same time. With multiple recordings, the OS can interleave the files in the same area of the disk to minimize seeks and maximize throughput. When playing back, though, it has no choice but to read from wherever the file happens to be stored, which entails more seeking and therefore lower throughput than a pure recording scenario. So ideally you'd like Sage to choose for recording a disk that's not already being used for playback.

It might also be handy to be able to specify that (for instance) my programs should be recorded to a different disk than my wife's programs, in order to minimize the chances of multiple simultaneous playback streams from the same disk.

[MrVining]

Seems like over time the current system would work great, because the longer it runs the more random the show size and space one each drive becomes (especially with more and more drives). At any rate the only other option for right now is a hefty raid card. Even then I wonder what kind of throughput a person would really end up with...

[Fuzzy]

a good PCI-e raid card should provide pretty good performance, especially in striped mode. I've even had decent results using windows built-in striping (well, not QUITE so built in, as it required some hacking to enable), though it did end up hitting a wall on throughput because I was limited by the simple SATA controller on the board.

The best option remains Sage actaully improving the selection system.

[MrVining]

I wont do striped mode, it's too risky. Besides a good raid card will write raid 5/6 just or near as fast as on board raid or software raid.

I guess I'm just going to have to get out the pen n paper and do some math. I'm looking to record upwards of 10-12 HD streams at once, 2-3 shows on playback, and one of these days I'll have to look into getting the com skip going.