Defrag Needed Even with 64KB Clusters

[jptaz]

Andy,

My experience has shown the same behavior. It is not every cluster, but it is usually 1024KB or greater chunks (16 Clusters at a time).

If Sage where to modify there Async Read/ Write Filter such that when a New File was created for a recording it could preset the file Size to the approximate recording length. This should be fairly easy since they know both the duration of every recording and the recording rate. If they under estimated the size you would end up with some light fragmentation at the end of the file, if they over estimated - perferable, they would just have to reset the length of the file. This is speculation on my part since I do not have detailed knowledge of the Frey Async File Read/Write Filter, but from my investigations into this on the standard dump filter it would work.

John

[stanger89]

I'm guessing it has more to do with multiple tuner simultaneously writing than free space. For example I've always set Sage to Use All, and my worst file is only 75 fragments. I've never had any trouble with pauses/fragmentation.

However I can imagine that with multiple tuners it would be possible to encounter circumstances where essenially every other cluster is written to by a different tuner/file resulting in huge fragmentation.

edit - I see that idea was alread brought up -/edit

FWIW, I was trying to see if IOMeter would be helpful in diagnosing problems. It may be but I can't tell from my setup. I created 4 workers, 1 reader and 3 writters, with a 1023MB alignment and 64k Transfer Request Size. My thought was that this should simulate to a degree a setup similar to John's. The result was that each worker averaged pretty close to 8MB/sec, more than enough for our purposes. And Random or Sequential IO made little/no difference for me.

I attached the config if anyone is interested.

[jptaz]

Stanger,

I downloaded IOMeter and I don't think it is a very valid test to simulate my situation. Correct me if I am wrong, but IOMeter creates one file on the disk and performs all the operations within that file. By doing this it bypasses the major problem, i.e. the filesystem over head of updating the MFT for the video file with a random cluster location then updating it. I suspect it is also bypassing the filesystem APIs which SageTV is no doubt using. This test suite does not really simulate writing 3 unique files that are being written and growing the file size. Every File write that happens to a file that is not preallocated needs to update a whole lot of information at once. From a raw performance stand point I do not think any of my disks have an issue as is evident from the HD Tach Test.

I think I ran the same test as you and I got a similar result of 7.95 MB read rate on one of my 80GB Drives. The file system overhead of dynamically growing files can not be underestimated.

I am not sure what causes a fragmented disk to cause this problem, but I doubt that the raw disk performance is an issue. Something else must cause the delay in reads while other writing several other files.

Hopefully some one will come up with a reason why a drive that performs so well can not handle writing at 1MB/Sec to 2 to 4 files randomly growing the files, while reading from 1 other file at the same rate.

John

[stanger89]

See that's what I'm not sure about. Yes IOMeter creates one file, but there's more to it than that. First it can create multiple "workers" which should simulate (reasonably accurately) multiple file or multiple user access. Then you can adjust the amount of randomness (100% Sequential to 100% Random access). So considering you can create a test with one or multiple readers and multiple writters, with anywhere up to 100% random access, it would seem that would be similar to what Sage is doing.

However like I said, when I tried it Sequential or Random didn't make a difference, so maybe it just won't work, or maybe I just don't know how to set it up....


I did a little reading about IO Meter and I may have made a better test. I changed the "burstiness" to 32 I/Os figuring Sage wouldn't be requresting 1 cluster at a time. I checked the reg and it looks like Sage uses 32 64k buffers.

[jptaz]

Very Strange...I got a pause last night briefly while watching a show. It was very brief, and only happened once.

What was interested was that my scheduled defrag only did two of the five drives. The drive the file I was playing back on was defragged, while the only active recording was on a completely seperate drive that was badly fragmented, but oddly enough the file that was being recorded only had 132 Fragments.

It also appears that I was wrong about SageTV putting many files on one disk. Last night all the recordings were going to different disks.

On a side note I dug up my WinTVCap Code and modified the File Write Filter and I was able to get it to preallocate a 4GB file and record to it. I then went into Defrag and Analyzed the disk the file I created on and it was one big contiguos file.

I am beginning to wonder if the performance characteristics change because I do have multiple drives. I am using every available IDE slot in my machine. So maybe a better simulation with IOMeter would to do your test, except with the write workers all accessing different drives. Since Accessing Two Drives on the Same IDE channel may be what is causing the problem. I would think the problem even with essentially 2 random Writes and 1 Random read on the same IDE channel would not cause this kind of pausing, but maybe there is something else going on that slows it down with that many different reads and writes on a shared IDE channel.

[stanger89]

Originally posted by jptaz
I am beginning to wonder if the performance characteristics change because I do have multiple drives. I am using every available IDE slot in my machine. So maybe a better simulation with IOMeter would to do your test, except with the write workers all accessing different drives. Since Accessing Two Drives on the Same IDE channel may be what is causing the problem. I would think the problem even with essentially 2 random Writes and 1 Random read on the same IDE channel would not cause this kind of pausing, but maybe there is something else going on that slows it down with that many different reads and writes on a shared IDE channel.

That should be easy to try, just open my test and set each worker to a different disk. I'd try but I've only got 2 disks, and both are on their own channel.

Maybe what you need is one of these:
http://cgi.ebay.com/ws/eBayISAPI.dll...116711343&rd=1

Cheapest 8-port IDE controller I know of. Then each of your drives would have it's own channel.

[salsbst]

Is it possible that you're saturating the IDE channel and that fragmentation is a red herring?

[stanger89]

Hate to say it, but you're also running on a VIA chipset motherboard, and VIA Chipsets are known for poor PCI performance, sometimes very poor.

[jptaz]

PCI Performance could be an issue. Defrag simply reduces the amount of traffic on the bus, by reducing the Over head.

It seems odd that PCI performance could play a part in this since searching on PCI Performance Issues with Via it appears to only come into play when pushing the performance limits with RAID or other high throughput. With 4 PVR 250 Cards, 4 IDE Channels (7 Drives), 10/100 NIC. It does not seem like this would be the same as a RAID type issue.

John

[stanger89]

Well with older VIA boards had trouble with all kinds of stuff. I remember comments about pops playing audio while accessing the HDD, HDD corruption etc. Not saying things haven't improved, but after my last VIA board, I'm rather biased.

[rrussell]

Somewhat off-topic: If I did get a newer IDE controller card, say ATA100 or 133, and put it in an older PC, will that PC then be able to see the more modern larger hard drives?

Like 160 and 200gb?

[Opus4]

Originally posted by rrussell
If I did get a newer IDE controller card, say ATA100 or 133, and put it in an older PC, will that PC then be able to see the more modern larger hard drives?

Like 160 and 200gb?

Yes. But, you may need to get some updates for Windows, depending on which version you are running.

- Andy

[kny3twalker]

since I was no help earlier I can at least suggest the Promise Card I am using

it works great there are versions with 2 serial ATA and 2 IDE as well as mine which is 4 IDE

[jptaz]

What is the Model Number of the promise card Kmy3tWalker?

I have a feeling it is my older 80 GB drives that are causing the problem not the Controller. My mainboard has a Integrated promise raid chip that they are all running at a good speed on, but the random access speed on the 80GB drive under bad random access conditons is ugly. Considering the worst the pausing got was once in an hour program, thats not bad for 3 year old 80 GB drives.

Also just an update. I forced things to start pausing again...this time it is definitely because of fragmentation...well at least it aggrivated what ever the cause of slow random access on my 80GB drives. I have doubts as to it being related to PCI or IDE congestion as all access was only on 1 hard drive.

I decided to do an experiment. I went in and deleted several shows from my e: that have been on the system for varying periods of time. I then started defrag and stopped part way through which left the disk horribly fragmented...actually more so than before. I then set 4 shows to record at the same time and since the e: drive had by far the most free space all 4 shows are recording to the one drive and it is at the 1 fragment per megabyte rate. In fact two of the recordings could not get stable ( they had 40+ different files and growing and was pausing horribly). I stopped them and started them again and this time this got a bit better since Sage Did not have a chance to clean up the partials before starting again.

This is an extreme example of what happens, normal levels of fragmentation created by SageTV does not immediately cause pausing it does take time, but I do think SageTV needs to get smarter about preallocating files or spec a min drive performance for multiple tuner cards.

Since it only took me 10 minutes to modify WinTVCap to preallocate files. Of Course it always preallocated them at 4GB a piece I am sure it will take a bit more time for Frey to add the intelligence to the write filter, but I am more than happy to just have it smart enough to restrict writing to one disk, especially when I had 28GB+ free only 3 other hard drives.

I definitely would prefer preallocation as it makes editing MPGs for making a DVD goes much faster. When I have edited commcerials on a fragmented file it takes for ever to skip through the file and it makes the disk sound like a popcorn maker.

Just my late night rants...maybe I will break down and buy a 250GB Drive and dump 3 of the 80GB Drives. If this doesn't solve the problem this I am looking at a new MB without a VIA chipset...who knows.

John

[kny3twalker]

I definitely would prefer preallocation as it makes editing MPGs for making a DVD goes much faster. When I have edited commcerials on a fragmented file it takes for ever to skip through the file and it makes the disk sound like a popcorn maker.

my 80 GB maxtor sounds like that before I added more RAM to the system

when it was still 256MB

and even then it did not quit completely but rather not near as much

the 250 GB drive I am using have the 8MB of cache

they seem to never do this at all
and since the 80 GB is not used as much
I rarely hear anything from it at all

maybe that helps
cause I feel your pain
about making popcorn

model number
does this mean I got to open my case?
it is in a cabinet

I can tell you it is not a RAID card
http://www.promise.com/product/produ...=87&familyId=3

[Opus4]

I don't know what the general impression of SIIG is, but I haven't had any problems with my ATA/133 card from them nor with my USB2/1394/NIC combo card. See SIIG's Ultra ATA & Serial ATA Products. Of course, I haven't benchmarked anything either.

- Andy

[stanger89]

Seriously, if you need a card, take a look at an old 3ware 5800 off ebay, they're really cheap, and even though it is a RAID card, you could just use it JBOD if you want. Each drive get's it's own channel so no master/slave issues.

[Opus4]

My problem with those 3WARE 5800 cards is that they appear to be ATA/66 only, plus you get a 5 year warranty with those SIIG cards, for less than the ebay buy-it-now price (SIIG prices at CompUSA).

However, the SIIG cards don't have 8 connectors, of course.

- Andy

[jptaz]

Originally posted by kny3twalker
my 80 GB maxtor sounds like that before I added more RAM to the system

when it was still 256MB

and even then it did not quit completely but rather not near as much

the 250 GB drive I am using have the 8MB of cache

Maybe more memory would help, I have been back and forth on this. I guess one possibility is that the MFT on all my drives is not cached since I only have 512MB of memory. Maybe I will borrow 512MB from another machine and do some testing.

In the middle of a major home improvement project so money is tight at the moment for none essential items.

John

[lonndoggie]

Disk fragmentation is a worry here. I've got two Seagate Barracuda SATA drives, one partitoned into C: and D:; the D: drive is -mostly- dedicated to Sage, but I also put all downloads there.

I have one PVR-250, and one MIT MyHD-120 HD tuner. The MyHD does it's capture thing to the F: drive, the other SATA Barracuda (not partitioned).

Looking at the drive Sage uses, I see it's 81% free space (which is typical), or 84GB. It also shows 31% total fragmentation, and 62% file fragmentation. Looking thru the file list, most (but not all) of the Sage recordings are fragmented (but not much--2 o 5 fragments). However, even small downloaded files are fragmented!

The F: drive is heavily fragmented, too; I do video file editing/transcoding there, and some files that are output of those processes have thousands of fragments (this confirming the theory of multiple threads writing different files increases fragmentation, where I'm doing a demux to create separate audio and video files).

And no, I haven't changed the cluster size to 64K. In this case, that might help a bunch.

I've been generally unhappy with disk performance. If, for example, I try to copy two large files from one drive the other simultaneously, instead of approximately halving the speed at which the transfers happen, as I'd expect if things are workiing well, it actually multiplies the time by a factor of 5 or so, and you can hear the drives buzzing as the heads do a cha-cha-cha getting and writing the files.

I have an ASUS P4P8X MB that has builtin SATA support.

Besides changing cluster size, any ideas of what I can look at to fine-tune my disk performance?

(btw, I had been having problems with Sage occasionally lagging, then catching up--defragging didn't fix that, but ReClock did).