64k Clusters

[KJake]

Yeah, so I took the advise of some people on the forums here and formatted my hard drive with 64k clusters. I figured that after using this new drive since Thanksgiving and having Sage crash a couple of times, I should defragment my hard drive - WRONG!

Turns out that Windows 2000 NTFS doesn't support defragmentation past 4k clusters...now what am I supposed to do??

[ji0005]

I use Norton Speed Disk, which comes with Norton Utilities, which can be found on Ebay for like $10.

[fidget]

Originally posted by KJake
Yeah, so I took the advise of some people on the forums here and formatted my hard drive with 64k clusters. I figured that after using this new drive since Thanksgiving and having Sage crash a couple of times, I should defragment my hard drive - WRONG!

Turns out that Windows 2000 NTFS doesn't support defragmentation past 4k clusters...now what am I supposed to do??

The reason for using 64K blocks is so you don't have to defagment the HDD.

[KJake]

That's what I thought too, but I was trying to watch this week's survivor episode and Sage kept crashing. So I did an analysis with defrag and that video file was the most fragmented (>60 fragments) on the drive (120gb).

I'm installing Windows 2000 Professional so that I can use Speed Disk since it won't install on 2000 Server (greedy bastards - it's the same OS).

Oh, this is fixed using Windows XP and 2003 Server. They fixed the Windows API used to defragment NTFS and normal defrag works the way it should. I'm not in the mood to upgrade to 2003 server tonight, so installing 2000 Pro on another partition will have to do.

[TunaBoo]

XP really is the way to go for a sage box because of how much more mature many drivers are (even when 2k and XP are the same driver, XP has so much more market share it gets much more attention).

[KJake]

I'm going to be moving into my house in 2 weeks and should be building a Sage only machine. Right now, Sage runs on my Web/Email/Shoutcast server too.

[sub]

Even if a file is fragmented, it wont cause it to crash. Fragmented files tend to be slightly slower to load, but they are still the same file. Fragmented doesn't mean its corrupted in any way. The Symptom you'd typically see with a fragmented video file is drop frames or stuttering, but only if you hard drive cant supply the data at the required rate.

I suspect you are seeing something else going wrong.

[KJake]

Well, I looked in my sage_tvXX.txt files and couldn't find anything that looked like a tell tale sign of a crash. So you think it was just a coinsidence that the most fragmented file on my disk was the file I was watching and after defagmenting my hard drive performance greatly impoved with Sage?

"When a file has become fragmented, it means that it is broken up into pieces on your hard disk. Imagine if you saved a file to your hard disk, and then saved another right after it. When you go to add more to the first file and then save it again, it no longer can fit in the space allotted, and must be split apart. When many files become fragmented, your hard disk performance is slower, and the danger of file corruption is greater. To fix fragmented files, you must defragment your hard disk."

[Bubster]

As sub said, fragmentation will only affect you if it lowers the rate at which Sage can get data off your hard drive to below the data rate of the video file.

If you are recording at DVD standard quality (3.25GB/hr), the data transfer rate required from the hard disk is about 1MB/sec, which is well within the capabilities of any hard disk (you should be looking at in excess of 40MB/sec with a decent 7200rpm drive).

However, figures like 40MB/sec can only be achieved with large, sequential (non-fragmented) reads. The reason for this is that hard disks are much slower at moving the heads to a different part of the disk than they are at reading from the next part. On a decent hard drive, this seek time is something like 9ms (milliseconds).

If you have 4K clusters, and all clusters are in random places, the drive will need to perform 256 seeks just to read your 1MB of data for a second of video. At a seek time of 9ms, this equates to 2.3 seconds worth of seeking. Add on your data transfer time, and your video is going to stutter very badly.

But with 64K clusters, you only need 16 seeks (assuming random placement or clusters) to read 1MB of data, which equates to 0.14 seconds. Even when you add on the (small) data transfer time, this is easily fast enough, and this is with a completely fragmented file.

Note: There will be another overhead for accessing the file system's file allocation structure, but this will be cached to a large extent, especially with a large cluster size. You also need to consider that, on a Sage server machine, you could be reading and writing different 1MB/sec streams at the same time, but this will still be do-able.

So what this all boils down to is that, if you are getting stuttering with 64K clusters, which is only fixed after defragmenting the drive, then you are suffering from one of the following:

1) Your hard disk's data transfer rate is too low.
2) Your hard disk seek speed is very slow.
3) Another program (or the OS) is making significant use of the disk at the same time.

(1) is very unlikely, although you should check that the OS is using the drive in UDMA mode, and not PIO mode.

(2) is also very unlikely. Even with a seek speed of 20ms (very slow), you should still be ok, and most drives are significantly better than that.

(3) is a distinct possibility. If you are running Sage on a machine with 256MB of memory or less, you may experience significant page swapping, which will destroy your hard disk performance. And there are plenty of other apps which can hog the hard disk if you let them. Try killing all unnecessary processes and see if that makes a difference.

As for the notion of fragmentation increasing the probability of file corruption, this is a spurious statement. Fragmentation can cause the impact of file corruption to be worse than it would be on a non-fragmented drive, but you should be looking at virtually zero corruption in the first place. The only time you should expect any corruption is when your machine reboots while writing to files and, even then, the possibility of this is vastly reduced with NTFS.

So the original advice of using 64K clusters is good advice. The only thing you need to be careful about is to put your system (and Sage) on a normal partition with 4K clusters (or close to this) and keep the 64K cluster partition for video files only.

- Neil.

[KJake]

Wow, thanks for the info on all of this.

The drive in question is a Maxtor DiamondMax 9. It's 120GB, ATA133, 7200RPM and I belive it has a 8MB cache.

I'm not sure if Sage was recording at the time or not. Sage is the only thing that touches that drive.

This is a separate video (G:) drive formatted at 64K, and Windows (C:) and Sage (D:) were formatted at default.

I have 512MB ECC RAM but only PC133.


Maybe there was just too much going on with my data bus or something? I have 3 hard drives in total on a standard IDE bus that handles ATA100 as the max transfer.

Sorry - I don't know a whole lot about hard drive transfer rates and such.

Maybe I can blame it on Windows 2000 Server? It's getting to be an old installation...and we all know Windows...but it just seems to me that the .15 release is less stable for me and I was pretty sure it was having to do with the fragmentation. Everyone proved me wrong though I think, so I'm not sure what to pin it on anymore.

I guess I'll wait to see how the new system will perform in a couple of weeks.

[stanger89]

What mobo (chipset)?

[KJake]

Gigabyte 7ZXE KT133 if memory serves.

[stanger89]

Could be related to your chipset. Do you have the latest drivers (may want to backup before updating)? I know when I had my KT266 board, it was not unheard of for HDD corruption to occur on VIA chipsets, especially under heavy loads, there was (is?) something wrong with PCI implimentation of those chipsets. You don't have an SB Live! too do you?

[ripple]

Originally posted by KJake
The drive in question is a Maxtor DiamondMax 9. It's 120GB, ATA133, 7200RPM and I belive it has a 8MB cache.

KJake,
Just as an FYI, I had a 120GB Maxtor DiamondMax Plus 9 and had it crash *twice* on me. First one crashed with my OS on it after a year so Maxtor shipped me a replacement which I added to my Sage system. After 30 days I started getting getting stuttering recordings and periodic "dead" spots in the recordings. I spent days trying to figure out what was going on and then the Maxtor crashed again completely. Maxtor replaced it again but now I only use if for "non-essential" storage .

You might want to run the Maxtor diagnostics (available on maxtor.com) and make sure it still passes all the sanity checks...

Personally, I'm sticking with Seagate from now on...
Kevin

[KJake]

stranger89: heh, I have a SB Live! Value with the Digital I/O 2 add-on.

ripple: I'll look at running the diags. I don't have a floppy in my server, so hopefully there is a windows version or ISO to download.

Maybe this is a sign of the Maxtor dying. I've seen Maxtor and Seagate drives die all the time. I only got this one because it was only $50 during the holidays. I personally only like Western Digital because I've only had one or two die since I got started in computers.

[stanger89]

Do you have a different soundcard you could try? I remember a lot of problems with SB Lives in VIA boards.

[KJake]

There is the on board sound (AC97, ick). I like to use the TOS (optical out) on the Live!. I updated my Via 4-in-1 drivers and rebooted to see if it would help. I'm at work for now, so I can't watch any video to test it out.

Looks like I have the most up-to-date BIOS.

One thing I've thought about is that the drive is set to operate at 133 and the chipset only supports upto 100...I haven't been able to find a utility like Seagate or WD that will change the mode to 33/66/100/133...does anyone know of such a util for Maxtor?

[Bubster]

I tend to find most claims about particular brands of hard disk being unreliable spurious at best. For every guy you meet who has lots of friends who had brand X fail repeatedly, there will be another guy who has the same story for brand Y, and brand Z, and so on.

That's not to say that, if you are having repeated problems with a particular brand, you shouldn't try switching. You probably should, as your problems may be caused by some weird compatability issues, but you shouldn't discount your previous brand as inherently unreliable. You may well find that, with your next motherboard choice, your new brand of hard disk starts becoming strangely unreliable.

If your machine is going to be in the living room, Maxtor and Samsung are very good choices, as they are known to be very quiet (esp. Samsung), whereas Western Digital drives are known for being a bit noisier.

SB Live cards are a complete nightmare. They pop up on more incompatibility lists than any other bit of hardware I've ever seen. You can solve a lot of the problems in various ways, including moving which slot the SB Live is in (e.g. SB Live & network card in slots 1 and 2 is a known problem area), but the simplest solution is to just throw the damned thing in the bin. The Audigy is much better in that respect, I should point out, just to be fair to Creative.

- Neil.

[KJake]

Yup, all valid points.

I have a real job now, and since I'm out of school I can finally afford new components and a new computer altogether. We all know the mix and match piece parts game, don't we?

I've wanted to move to the Audigy for a while so that I can get 5.1 (6.1 or 7.1) from the computer rather than the "surround" sound of the Live.

I was looking into Samsung drives for my next Sage box exactly for those reasons - I'd like to build a really silent PC - anything should sound quieter than my current server system.

[kny3twalker]

Wow my maxtor hadrddrive is the noisiest piece of crap I have ever heard
Man before I added another 1/2 GB of memory
It sounded like it was just making it self some popcorn when doing anything
pop, ppop,pop ,pop, pppopp, pppop, pop, etc......
noisey as hell
I like western, not very loud compared with my maxtor and reliable