Stutter from WHS drives?

[Tokendude]

I'm having a problem I haven't been able to resolve and hope that someone here can offer some light.

I've got what was, until a week ago a very stable, long running, very basic system. A server box running WHS 2003 5.2, SageTV v 6.6.2.218 in service mode, a single HD200, and a gigabyte network switch. The server box is a quad core processor with 5 pool drives (64kb sector format) 6.5 TB total, 4 gigs memory. Tuning is from 2 HD Homeruns recording over the air digital broadcasts. Java heap: 147/194/1065. I'm not running any special background programs or shells. It's a plain vanilla install that's been bulletproof for the last 4 years or so.

Recently some recorded shows began stuttering on playback, and menu response WHILE A VIDEO WAS PLAYING became sluggish at times. Note that DVD "rips" (just copies of *.VOB files, not recoded files) still played fine. At first I thought it might be a network issue, however I believe I have ruled this out. I say that because last night I was able to get a Nov 2012 recording of Bones to play flawlessly while a Feb 2013 recording of Bones consistently stuttered. It did NOT stutter at the same place however, so the actual recording is ok. I went back and forth several times between the two recordings and the newer one stuttered while the older one did not.

I've ruled out the HD200 as I have several and switching them out (still only having 1 hooked up at a time) shows no change.

I thought I might have a hard drive beginning to fail, but running all 5 pool discs through SPINRITE showed no errors or problems. On the pool, there is 6.5TB total with 2TB free. Analyzing the discs (but not defragging since you can't do that) shows fragmentation from 20 to 40%.

I'm lost as to the cause. I can speculate that maybe it could still be a hard drive problem, but to isolate I guess I will need to remove 1 hd at a time from the pool and check performance afterwards.

I have not installed any updates recently to the server, windows or otherwise so that did not cause the problem. The only thing I did do recently is delete a very large number of recorded shows (about 1TB worth) as part of a system cleanup.

Now that I think about it, the only other symptom I've seen is that when I've copied a few ripped dvd folders onto the server something happens to the transfer and it fails stating that the network name connection has been "lost". This is an operation external to SageTV. This is why I thought it might be a network problem, but as I said above, I believe that has been ruled out.

Anyone have any other ideas to try if isolating the pool drives doesn't work?

[waynedunham]

I have a somewhat similar setup. Gigabyte Mobo (EP45-UD3P), Quad Core CPU, 3 1tb pool drives, 500gb system drive, Gigabit lan. I have a plethora of tuners.... 2 HD-PVR/STB's, 2 HDHR (1 original HDHR dual tuner), 3 HDHR (1 HDHR Prime, no CC).

The only thing I see in your message similar to something I did once is the deletion of a bunch of files (most on a single drive of the pool). When drive balancer kicked in the SageTV system was just about unusable with my HD100 extenders. Once drive balancer finished all was well again.

The only other time I experience issues like that is when WHS starts backing up my other 3 PC's (old Sage Server that still is running, my main pc, and a laptop).

So what may look like a network issue (I thought the same thing when it first happened to me) may just be the server running out of "gas". One thing I haven't checked, since it happens very rarely, is the program affinity for SageTV Service, and maybe lowering the affinity for backup and/or drive balancer.

[BobPhoenix]

Originally Posted by waynedunham

I have a somewhat similar setup. Gigabyte Mobo (EP45-UD3P), Quad Core CPU, 3 1tb pool drives, 500gb system drive, Gigabit lan. I have a plethora of tuners.... 2 HD-PVR/STB's, 2 HDHR (1 original HDHR dual tuner), 3 HDHR (1 HDHR Prime, no CC).

The only thing I see in your message similar to something I did once is the deletion of a bunch of files (most on a single drive of the pool). When drive balancer kicked in the SageTV system was just about unusable with my HD100 extenders. Once drive balancer finished all was well again.

The only other time I experience issues like that is when WHS starts backing up my other 3 PC's (old Sage Server that still is running, my main pc, and a laptop).

So what may look like a network issue (I thought the same thing when it first happened to me) may just be the server running out of "gas". One thing I haven't checked, since it happens very rarely, is the program affinity for SageTV Service, and maybe lowering the affinity for backup and/or drive balancer.

What you are talking about here is exactly why I Virualized my SageTV server separate from the WHS. They still run on the same physical PC but in separate VMs with WHS limited to one virtual core and three virtual cores for the SageTV server in ESXi. I also have an unRAID server on that box in a VM as well. WHS is now used for backups only.

[Tokendude]

By watching which files stutter, either due to bad recording or bad playback, I believe I've narrowed it down to a 2TB WD AV-GP hard drive. It passed every hd diagnostics test I threw at it, but only files stored to this particular drive in my pool give me problems, and the problem only occurs when there is heavy activity on that drive (ie more than one file being accessed at the same time). My three 1TB WD hard drives are RE3 drives and the 1.5TB hard drive is a WD black.

While I didn't realize it when I bought it, although the AV-GP is designed for video work, it apparently doesn't contain any error correction. If something happens during recording or playback, this drive is just supposed to keep on going and ignore the error so that it can get back to either playing or recording as soon as possible.

To prove this drive is at least part of the problem, I've ordered a replacement drive that is a WD 2TB RE4 drive. Twice as expensive as the AV-GP, but if it solves the problem then it will be well worth it. Momma doesn't like it when her soaps get messed up, and when Momma ain't happy, nobody's happy...

I'll update my findings mid week.

Roy

[gplasky]

Originally Posted by Tokendude

By watching which files stutter, either due to bad recording or bad playback, I believe I've narrowed it down to a 2TB WD AV-GP hard drive. It passed every hd diagnostics test I threw at it, but only files stored to this particular drive in my pool give me problems, and the problem only occurs when there is heavy activity on that drive (ie more than one file being accessed at the same time). My three 1TB WD hard drives are RE3 drives and the 1.5TB hard drive is a WD black.

While I didn't realize it when I bought it, although the AV-GP is designed for video work, it apparently doesn't contain any error correction. If something happens during recording or playback, this drive is just supposed to keep on going and ignore the error so that it can get back to either playing or recording as soon as possible.

To prove this drive is at least part of the problem, I've ordered a replacement drive that is a WD 2TB RE4 drive. Twice as expensive as the AV-GP, but if it solves the problem then it will be well worth it. Momma doesn't like it when her soaps get messed up, and when Momma ain't happy, nobody's happy...

I'll update my findings mid week.

Roy

I hope it's not your SATA cable for that drive that is the issue.

Gerry

[Tokendude]

I've found at least part of my problem. The WD AV-GP drive's controller had dropped from DMA access mode to PIO access mode. A drive in PIO mode is useless for media streaming or recording because of the limited bandwidth.

While in PIO mode the offending drive would only sustain a throughput of around 3,000 KB/sec as viewed in the WHS console Disk Management tab. Resetting the access mode to DMA increased the throughput to between 60,000 and 70,000 KB/sec. That is comparable to what the other drives were showing and all stuttering problems have been eliminated.

Of course, something had to cause the system to drop down into PIO mode on that particular drive. It could have been a power fault (my UPS only works for ~30 min before failure), or an actual problem with the hard drive. I'm going to leave it in the system for a while and see if it drops down again.

For those interested in reading about DMA vs. PIO and how to reset easily here is a link. It uses a Visual Basic script to reset all channels to DMA access mode. I'd suggest downloading the script instead of just running from the web page as it's a handy tool to have onhand.

I'm a happy boy again

[waynedunham]

Yeah, having the drive drop back to PIO mode would probably do it.

[KryptoNyte]

Did those AV-GP drives suffer from the same head parking problems that the straight up green drives suffered from?

I just got through RMA'ing (3) WD Green 2TB drives that I had in RAID-0 on my Sage Server. They were really starting to frustrate me, an not even (2) years old. I pulled all three drives and tested them individually, and all (3) failed.

Western digital was good enough to replace them with (3) 3TB green drives ... basically the same technology

[Tokendude]

If you're talking about the head parking/high cycling issues, no I do not believe so. I understand that problem is also resolvable by a program called WIDDLE, but since I have no experience with plain green drives I can not say for sure.

For me I'm sticking with the more expensive but rock solid RE drives from now on. They run cooler than the other WD drives (10 degrees F vs. black drives and 5 degrees f vs. AV-GP drives). They also last forever. Not exactly what they were designed for, but still a good value since WHS does not fail gracefully when using their Pool drive function.

[KryptoNyte]

I had already completely swapped out all WD drives with Seagates, but yeah, I ran WDIDLE3 to increase the park time on all (3) of the new drives. I can only hope that works as time passes - 21 months is not an acceptable time to failure for any hard drive bought with money.