Halt Detected in Recording - repaired

[camTexas]

I ran in to a problem using the Diskeeper trial that was supposed to be WHS compatible. Needless to say I had to rebuild the server and lost several of my ripped DVDs and recorded shows due to file corruption problems. I'd also be interested in a reliable defrag solution for WHS, although I guess now, having reformatted all my drives (6TB) during a clean install, it should be a while now before I have an issue again.

[gplasky]

If your drives were formatted with 64k blocks it shouldn't need defragmenting. I've run WHS for almost 2 years without a defrag. If you decide to defrag it must be a tool specifically designed for WHS.

Do you have folder duplication or shadowing turned on? Do you have your Recording directory configured for a share? (Not a drive)

Gerry

[Skiier__Dude]

Yes, my drives are formatted in 64k blocks.

My system drive (C is on a 20Gig partition and the rest of my drives are in a shared drive pool (D. As I have been adding drives to this pool I have been formatting them in 64k blocks.

I do not have folder duplication turned on for the Sage recording directory (just for music and photo directories).

[voicu_n]

Originally Posted by Skiier__Dude

Hi Big Gerr,

Thanks for the reply. In fact, when I go into the WHS Console and look at the Add-Ins section, Sage does NOT appear. The Sage service appears in the console and I can start and stop it from there.

I'm going to do as you have advised and get rid of my installation again and see what happens. Like you say, may or may not be the smoking gun, but worth the time doing it right.

Hey Skiier__Dude are you sure you are installing the Sage for WHS add-in ? the one from here : Download SageTV Media Server (V6.6 for Windows Home Server)
and not the one for regular windows here :
Download SageTV Media Center (V6.6 for Windows)

Seemed strange to me you dont see it in the add-in section, after installation ...
I was having same problem too , looks like for me was the lack of space on the recording hard drive ... Just emptied the drive and waiting to see the results now .

[martin]

Originally Posted by gplasky

If your drives were formatted with 64k blocks it shouldn't need defragmenting.

How would 64k blocks prevent my file system from becoming fragmented? AFAIK every file system can become fragmented. If fragmentation causes choppy playback, why couldn't it also affect recording?

[Skiier__Dude]

Originally Posted by voicu_n

Hey Skiier__Dude are you sure you are installing the Sage for WHS add-in ? the one from here : Download SageTV Media Server (V6.6 for Windows Home Server)
and not the one for regular windows here :
Download SageTV Media Center (V6.6 for Windows)

Yes, when I re-installed Sage a couple of weeks ago I did it via the Add-In method using the .msi file and I now see it in the Add-Ins section.

So when I have Windows Disk Defrag tool do an analysis on each of my drives, it's telling me that the files on each of the drives are about 90% fragmented.

May have to investigate a safe way to defragment these drives using a safe WHS tool.

[Skiier__Dude]

Further updates -- I've installed PerfectDisk and had it run an analysis and the disks are not as defragmented as the Windows tool outlines. Fragmentation is still high, but is around 20% not the 90% the Windows tool reports. I am going to have it run a defragment overnight to see if things improve.

Also, changing the drive reference on the recording directories from UNC paths to direct drive letter paths (as the Sage support tech recommended) has had no impact -- I've seen just as many if not more Halt Detection errors tonight -- 18 of them while recording a hockey game tonight.

[mickp]

Hey there,

It's probably a stupid question, but I don't see any mention of checking the windows system event log (eventvwr.msc) for disk or ftdisk related errors.

Errors like:

Event Type:	Warning
Event Source:	Disk
Event Category:	None
Event ID:	51
Date:		13/11/2009
Time:		11:33:12 PM
User:		N/A
Computer:	HAL
Description:
An error was detected on device \Device\Harddisk5\D during a paging operation.

and

Event Type:	Warning
Event Source:	Ftdisk
Event Category:	Disk 
Event ID:	57
Date:		13/11/2009
Time:		11:32:57 PM
User:		N/A
Computer:	HAL
Description:
The system failed to flush data to the transaction log. Corruption may occur.

can definitely cause the type of problems that have been reported. A chkdsk /f will not necessarily discover or repair these errors

Top Tip: Always perform a full format rather than a quick format on drives. Karma/Sod's Law dictates that every time you do a quick format on a drive there will be a physical problem that will not manifest until you've been using the drive for a while.

Also re the change from unc to local disk paths, I've experienced problems with using unc paths in the past (still use pc clients).

If you need to keep support for unc paths but don't want local traffic to be dependent upon the physical interface, you can install the Microsoft loopback adapter, set it to automatically find an IP address and it will automatically allocate a 169. address. Configure the priority of the network adapter to be highest in Control panel/Network Connections/Advanced/Adapters and Bindings/Connections. Also make sure that it's bound to 'File and Printer Sharing for Microsoft Networks' and 'Client for Microsoft Networks'. Once you've done this, you will be able to communicate with the rest of the network via your physical nic but to your own machine via the loopback adapter. You can verify this by pinging your machine's netbios name from your machine. It should resolve to a 169. address.

LMK if this might be useful and I'll add detailed instructions.

Here's a pic



Mick.

[Skiier__Dude]

Thanks for the reply Mick.

I no longer need UNC paths so I've removed them.

I ran PerfectDisk over night and it has defragmented all of my disks. It reports that the system went from 25% fragmented to .01% and now reports performance of the disks as excellent.

I took at look at the Event Viewer. It reported some Atapi errors during the defragmentation process (which I suspect is likely normal as the disk would be very busy). Otherwise, there have been no I/O type system errors reported.

A recording is running now for the next few hours and I will see what happens with the errors.

[mickp]

Originally Posted by Skiier__Dude

I took at look at the Event Viewer. It reported some Atapi errors during the defragmentation process (which I suspect is likely normal as the disk would be very busy). Otherwise, there have been no I/O type system errors reported.

A recording is running now for the next few hours and I will see what happens with the errors.

I don't think that you should expect atapi errors. What were they?

Mick.

[Skiier__Dude]

The atapi errors that were reported (only during the defragment) were Controller Errors (device detected a controller error on port 1) -- the only time these errors were ever reported was during the defrag process.

Now however, I'm getting a new error in the past day that indicates disk 2 has a bad block (first time this error has been reported as well). Wondering if there may be a disk going bad here in the background that is causing a bunch of I/O errors, but WHS has not picked up?

I am in the process of having WHS remove the disk from service and I will then reformat it and run the Western Digital diagnosis tool.

In the meanwhile, I also have a new HD-PVR on it's way to see whether that device is causing the issue. At this point, Sage is completely dead and unusable. Lots of maintenance this weekend ahead.

[mickp]

Originally Posted by Skiier__Dude

The atapi errors that were reported (only during the defragment) were Controller Errors (device detected a controller error on port 1) -- the only time these errors were ever reported was during the defrag process.

Now however, I'm getting a new error in the past day that indicates disk 2 has a bad block (first time this error has been reported as well). Wondering if there may be a disk going bad here in the background that is causing a bunch of I/O errors, but WHS has not picked up?

I am in the process of having WHS remove the disk from service and I will then reformat it and run the Western Digital diagnosis tool.

In the meanwhile, I also have a new HD-PVR on it's way to see whether that device is causing the issue. At this point, Sage is completely dead and unusable. Lots of maintenance this weekend ahead.

Damn, that's not good news.

I'd still be inclined to think that a controller error is something to be concerned about. There's just not room for things to go wrong like this in a disk subsystem regardless of how busy it is. Rationalising that it's only happened during a defrag seems like commenting that a car only runs rough while it's being driven.

Re: the WD diagnosis tool; my advice is not to pay too much heed to the S.M.A.R.T. counters when they read 0 errors. I've successfully RMA'd several drives which could not be successfully formatted and were making that "shaking a can of spraypaint" sound yet had all of the S.M.A.R.T counters reporting zero errors. D.U.M.B error reporting imho . A full destructive write test should do the trick tho.

Good luck.

Mick.

[Skiier__Dude]

Thanks. I've just gone through that this weekend. WHS started reporting a bad block in one of the drives late last week. I initiated the removal of the drive on Friday night and it took until Sunday night for the data to be copied off the drive. The system seemed to remove all of the data from the drive, but didn't take it out of the drive pool.

I ran the WD-DLDIAG extended test and I can see there is seriously something wrong with the drive. The extended test indicates it will be 120 hours before it completes it's diagnosis (on a 500G drive). I let the diagnosis run for 12 hours before I shut it down. I run the same test on another 500G drive and it reports it will be done in <2hrs. This particular drive also shows up in the WHS Disk Management tool at about 5-10 degrees C warmer than the adjacent drives. Going to RMA this disk and get it out of the pool. Now the trick is to seee whether the recording errors go away.

Note the Atapi errors ended up being a dirty/unreadable DVD left in the tray. Forgot that I had tried to rescue one of the kids DVDs and didn't remove it.

Interesting that even though there seems to be a bad disk, the WHS disk management tool reports everything as healthy, yet the system tools report a bad block in a disk. You'd think there would be a better error reporting mechanism in WHS?

[Skiier__Dude]

I hate to even suggest it, but.....so far so good. Removed that 500G disk out of the pool and fired off 3 test recordings. No Halt Detected errors so far (normally I'd be at something like 40+ errors by now). Not declaring myself completely out of the woods, but it looks a load better than a couple of days ago.

[dooferlad]

Just to add my 2p I just had my Sage box die on me and I had been having a lot of these errors. Hard disks remap bad sectors on the fly, so as soon as you see Windows report any buy a new disk quick (or get it replaced under warranty depending on your options). I hadn't spotted any disk problems, but I probably should have been taking at least a passing look at various error logs and running a SMART monitoring program.

[cychou]

Originally Posted by martin

The halt error was happening more frequently on my system until I cleared out my Video Recording directories. I have four 250GB recording drives, but I had so many recordings I didn't want to delete that Sage was left with only one drive to work with, and it became badly fragmented. Three of the drives had zero space available, so my scheduled weekly defrag with Diskkeeper failed.

This weekend I just finished expanding my linux file server to a capacity of 30 disks to give me more archive space. After I moved 400GB of recordings off to a Media Centre Import directory located on my file server and defragged the rest of the recordings, it has been running smooth for a few days now.

I noticed before that Sage will record reliably for days, but as soon as I stress the server by playing back a recording on my HD200 I got the halt error immediately. It seems the HD-PVRs are sensitive to I/O waits, almost like they have small buffers.

Anyone else observe this?

Thanks for the tip, I will give that a try.

[Skiier__Dude]

Three more days have passed and things are still working 100% so I think all is now well in Sage-land.

Lesson learned: Halt Detected errors were showing up because the recording devices were unable to write quickly enough to a failing disk. I suspect their buffers filled as they were unable to send the data. There could be a better error message reported as the standard message certainly pointed the blame and effort at the recording devices rather than the I/O.

Anyhow, hope this thread is helpful to someone in the future.

[wayner]

I too have started getting these errors recently. I little while ago I added an additional 1.5TB drive to my storage pool to bring it to 6.6TB. While recording basketball or hocky games I often get from 15-20 halts ina a 2-2.5 hour recording. These errors happen on both of my HD-PVRs at the same frquency but not from my OTA tuner nor my PVR-150 tuner. That would substantiate the disk access problems as the hardware is somewhat different.

These errors don't actually seem to be affecting the recording - at worst there are microstutters or short breaks in the recordings. Is it possible that these were always there but they were less noticable before Sage added the System Messages function.

[lamanmi]

I had the exact same problem when I added a WD 1.5 TB drive. I actually had two exactly the same, purchased at the same time, one on the Sage server and one on a second computer that serves as a offline backup of the main drive.

I did a Quick Format at 64kb sectors. The one in the Sage server would have problems when recording and watching a show at the same time from that drive (both delays while watching and the recording faults). I switched the drives and it the problem went away. Although both drives seemed to work fine from the OS perspective, there was something about one of them that caused the problem in Sage.

So from my experience, you may want to try:

- Getting replacement drive (not all drives created/tested equal)
- Do NOT use quick format to format the drive (saw this also in this thread)

[theguru]

I thought I'd mention this in this thread.

I was seeing these halted mesage as well on my new external 1TB disk connected by USB. I decided to move this drive out of the enclosure and just attach it internally via SATA and no more halts.

I figured a couple HD streams to this drive would probably overload the bus.