Hard Drive Utilization

[wolfpackmars2]

I have a general question concerning how Sage handles a "busy" hard drive.

This question comes up as I have increased the number of showanalyzer processes from max 2 to max 4 at a time. At the same time, I have begun using VideoRedo again, and didn't have SA set to create the VideoRedo project files - so I deleted the comskip files for several shows to prompt SA to create new comskip and VideoRedo files.

I noticed that CPU utilization on the PC was very low, even with 4 SA processes running at the same time - yet it was taking a long time to get through all the shows. SageTV is currently recording all shows on one hard drive - the largest one, even though 2 hard drives are configured for saving shows. I guess Sage uses the hard drive with the most free space (1TB vs 250GB)? Anyways, there was one episode on the other hard drive that I wanted processed. This episode was processed in a FRACTION of the time it has taken SA to analyze any single episode on the other hard drive. Then it dawned on me that the problem wasn't CPU resources but hard drive resources. Even with a SATA-II hard drive, reading 4GB worth of data across 4 different files on the same hard drive is going to be intense.

My next thought is - if SA is utilizing the hard drive at or near capacity and Sage tries to record a show to this hard drive and finds it busy, how will Sage deal with this?

In general, I don't think it will be an issue - once I get through with this push to add videoredo data for the current episodes, I think that the headroom on the hard drive will easily handle 4 shows while they are recording. I just want to avoid any possibility that sage might start dropping frames when the hard drive can't accept data as fast as Sage can feed it.

[GKusnick]

If you have only two recording drives, why would you want to analyze more than two shows at once? Seems to me that for best performance, you want to analyze just one recording per drive at a time.

It's like trying to vacuum three or four rooms with just one vacuum cleaner. In theory you could do it in parallel, running back and forth taking a few strokes here and a few strokes there, but in practice it's much more efficient to stay in one place and finish off each room before moving on to the next.

The read head on your drive is like that vacuum cleaner. If disk I/O is the limiting factor, then making it run all over the place from file to file and back again is not going to speed up processing; quite the contrary. Let it finish analyzing one file before asking it to start on another.

[Beefcake550]

This is almost true. non mpeg2 HD commercial skipping is a CPU limited problem. All other commercial skipping is a drive limiting problem. So, if you are skipping through HD-PVR recordings, limit it to 2-4 @ a time. I use 2 on my quad core. If you are not using an HD-PVR, then you should limit it to 1 at a time for best overall performance.

You could also use Sage Job Queue to not start any new processing when a client is connected and watching TV/media.

I have a HDHR and a HD-PVR, so I use SJQ to tell if it's MPEG2 or not. If so, do one at a time, if not, do one at a time.

[stanger89]

Originally Posted by wolfpackmars2

My next thought is - if SA is utilizing the hard drive at or near capacity and Sage tries to record a show to this hard drive and finds it busy, how will Sage deal with this?

To answer this question, AFAIK Sage just "writes" to the drive, it doesn't know or care how busy the drive is, it just calls to the system and says to write data and expects it to get there.

This is one thing that intrigued me recently. I've been looking into building/switching to a linux server, and while researching linux filesystems I came across XFS. XFS has a very interesting feature, "Guaranteed Rate I/O". The idea is an app can reserve a certain amount of bandwidth to the file system and that XFS will guarantee it.

It would be really sweet if Sage implemented that sometime, having Sage reserve say 15Mbps read/write to the file system for an HD PVR recording, or 20Mbps for an ATSC recording, and then you could pound the drive with whatever you want beyond that without affecting Sage. Of course it would only work in linux as that's the only place that filesystem is available.

I think showanalyzer can be limited to reading at a certain rate, I haven't found such a setting for comskip yet.

[briands]

Originally Posted by stanger89

To answer this question, AFAIK Sage just "writes" to the drive, it doesn't know or care how busy the drive is, it just calls to the system and says to write data and expects it to get there.

This is one thing that intrigued me recently. I've been looking into building/switching to a linux server, and while researching linux filesystems I came across XFS. XFS has a very interesting feature, "Guaranteed Rate I/O". The idea is an app can reserve a certain amount of bandwidth to the file system and that XFS will guarantee it.

It would be really sweet if Sage implemented that sometime, having Sage reserve say 15Mbps read/write to the file system for an HD PVR recording, or 20Mbps for an ATSC recording, and then you could pound the drive with whatever you want beyond that without affecting Sage. Of course it would only work in linux as that's the only place that filesystem is available.

I think showanalyzer can be limited to reading at a certain rate, I haven't found such a setting for comskip yet.

Of course, that may be tying up drive "resources" unnecessarily, since Sage is not always recording to each drive. What would be neat would be a priority setting for drive throughput, much the same as processor priority that windows manages. That would likely be a significant change for windows... but it seems like it may be a more common issue as media files become larger and drive speed increases seem to have stagnated.

[stanger89]

Originally Posted by briands

Of course, that may be tying up drive "resources" unnecessarily, since Sage is not always recording to each drive.

I'm not following, or I think you misunderstand...

What would be neat would be a priority setting for drive throughput, much the same as processor priority that windows manages.

This is basically exactly what XFS allows. The program can reserve a certain bandwidth while it needs it (I assume only while it needs it), so in theory with XFS, when Sage is about to record a 15Mbps HD PVR recording, it could call into the XFS file system, and reserve itself 16Mbps worth of bandwidth for that file, thus since XFS has a guaranteed I/O function, comskip or file copies, or anything else would be unable to interrupt Sage's bandwidth. Sage could do the same thing for each record (write) or playback (read) operation, and theoretically we'd never have recording or playback glitches again, at least not due to HDD bandwidth issues caused by other apps.

[briands]

Originally Posted by stanger89

I'm not following, or I think you misunderstand...



This is basically exactly what XFS allows. The program can reserve a certain bandwidth while it needs it (I assume only while it needs it), so in theory with XFS, when Sage is about to record a 15Mbps HD PVR recording, it could call into the XFS file system, and reserve itself 16Mbps worth of bandwidth for that file, thus since XFS has a guaranteed I/O function, comskip or file copies, or anything else would be unable to interrupt Sage's bandwidth. Sage could do the same thing for each record (write) or playback (read) operation, and theoretically we'd never have recording or playback glitches again, at least not due to HDD bandwidth issues caused by other apps.

Sorry... did not catch the "while it needs it"

[stanger89]

Well I did just imply it the first time

[Stuntman]

I wonder why Sage can't or doesn't use it's own scheduler and assign recordings to drives.. Much like it assigns tuners based on availability, I would think it could check all your drives that have recording space available and then check it's schedule and assign different recordings to different drives based on space and number of concurrent recordings. I've often wondered why sage would try to put 3 shows on my drive "G" at the same time when my drives "D,E,F" all have free space! Couple that with SA or Comskip, and perhaps even be watching one or more other items on that same drive and you see that drive "G" quickly gets swamped! While Sage can't control what we watch, it should be able to intelligently schedule it's recordings to use the drives available more effectively.

Perhaps there is something in there already, but from what I see, it doesn't appear to be the case.