HD-PVR Stops Responding ~Once/Day

[eric3a]

There seems to be a lot of speculation and hacking here, but not a lot of methodical testing.

I'd argue I've done some fairly methodical testing in the last 6 months, taking each supposed cause as discussed in the various forums, and testing one after the other.
A few others that I trust for thoroughness and logic also did the same things and we pretty much independently reached the same conclusion: There is some voodoo involved!
BTW, all your items were included in my tests, and none were identified as single culprit.

More seriously I now believe that there is a likely combination of (at least) 2 things, but you're right that it's speculation because I can't test either:
-Drivers/HDPVR firmware not installing properly and/or not being compatible with your HD receiver. It may be that the firmware can get corrupted on its own during power glitches or resets. Pretty much one of the few explanations for the "it used to work, I haven't changed anything and now it locks-up all the time" followed by "reinstalled same driver and it works again", which has been reported a few times. Some on BTV forum had working units for a while and took many re-installs of the same driver to get back to working again. As if suddenly one of the installs "took".

-Poor signal / weird signal coming from the HD receiver during some programs.

For info, my 2 HDPVRs are now running very stable with only 1 lock-up on 1 unit in almost 4 weeks. I used ot have daily lock-ups. My auto-reset script via SageAlert caught it and took care of it, so really I now have very reliable HDPVRs.

I am now running 1.0.5.301, the VERY SAME driver that did not work when I first starting using my HDPVRs.

I am running a PCI NEC usb card (made no difference on its own), running the units as cold as possible via a monster fan on their underside (no difference on its own) and even ran them open for a while, IR windows taped (no difference on its own), E1 and D2 unit (found no reliability difference between my 3 units), HD receiver locked at 1080i (no difference...), using a UPS and a resetable power supply (no diffference...), increased java memory (no difference...)... You get the picture.

At least your issue seems easier to pinpoint since the lock-ups happen during playback, so there's likely a bottleneck somewhere. Likely caused by playback.
What does the Task Manager show?
Any window system events in the system event viewer?
What happens changing playback codec?
What happens changing playback mode (overlay vs 3D)?

If I were you I would concentrate on the playback side of things...

Good luck
Eric

[jerryt]

Originally Posted by ewelin

I had tried a couple premptive power cycles earlier this week and I still had lockedups shortly after doing them. For example, yesterday I power cycled the units at 7 as nothing was recording and they hadn't locked up for 22 hours. The recording at 8 locked up twice.

@ Ewelin
Were you able to get drivers 1.0.5.301 installed and working?

[jerryt]

Originally Posted by martin

There seems to be a lot of speculation and hacking here, but not a lot of methodical testing. Between all of us we should be able to narrow this down.

There has and is a great deal of effort toward troubleshooting these lockups. It is possible that there are multiple factors that come together to create these lockups. The term "VooDoo" simply impies that we do not know how the logic of how these factors come together. Clearly there is a list of corrective measures which have worked for other people.

I would like to see a data base complied with which equipment works with which drivers. For example;
1.0.5.301 drivers with a Dishnetwork VIP211K receiver (RCA Audio, no audio sync issue and no HD-PVR lockups)

Maybe if all the data were complied in one place the trends would be clearer to identify.

In troubleshooting many systems I have learned that sometimes a direct understanding of an unknown problem can not be formulated. But after "throwing parts at it" the problem can be discovered and the direct understanding reversed engineered (Call getting lucky).

[ewelin]

Originally Posted by jerryt

@ Ewelin
Were you able to get drivers 1.0.5.301 installed and working?

Not yet.... if i wrap up the project I'm working on today quickly I may get to it tonight. I hope! LOL

[larryf]

Originally Posted by eric3a

-Drivers/HDPVR firmware not installing properly and/or not being compatible with your HD receiver. It may be that the firmware can get corrupted on its own during power glitches or resets. Pretty much one of the few explanations for the "it used to work, I haven't changed anything and now it locks-up all the time" followed by "reinstalled same driver and it works again", which has been reported a few times. Some on BTV forum had working units for a while and took many re-installs of the same driver to get back to working again. As if suddenly one of the installs "took".

Eric

I believe you have identified the problem... The firmware seems to get corrupted by heat or other glitches, and would explain why many have reported "its working again' after a driver reinstall that did not work before.

One of the previous firmware/driver updates even claimed to "fix" an issue where the HD-PVR firmware can become corrupted and render it unusable.

Is there any way to get a checksum of the installed firmware on a HD-PVR? I'll bet Hauppauge has something that can check the firmware, or at least they have a tool that can reinstall firmware without reinstalling drivers. They surely don't run HWCLEAR every time an HD-PVR is reflashed and sent back to the field.

Larry

[tvmaster2]

Originally Posted by martin

Has anyone found a way to reproduce the bug? This is the first step to solving this problem. I haven’t tried all of these myself, but so far, none of the following suggestions have prevented lockups:

1. Do a clean install of the drivers
2. Revert to 1.0.5.301 drivers after using hwclear
3. Keep the HD-PVR cool
4. Don’t use SPDIF for audio, use RCA
5. Don’t use hardware versions earlier than E1
6. Defragment your hard drives
7. Lock component output to 720p
8. Use a NEC PCIe USB card
9. Filter power through a UPS
10. Increase Java memory

There seems to be a lot of speculation and hacking here, but not a lot of methodical testing. Between all of us we should be able to narrow this down. I have noticed with my system, the halt error occurs during playback. If I’m not watching a recording, the error doesn’t happen. I tried to reproduce this by recording with all four of my tuners simultaneously, and watching three recordings at the same time. When I do this, the halt occurs immediately. My two HD-PVRs will not start working again until I plug them into different USB ports.

What other facts do we know?

It's a buggy bit of hardware, sadly. Mine locks up even without Sage - just open the Arcsoft Theater software and open the record window (and you don't have to hit "capture") - eventually it will crash. So that gets Sage off the hook.
What's the point of NOT using the spdif audio in? Why would I want to record HD video and not 5.1 audio?
Same goes for 1080 - that's the HD spec, so this thing should be able to deal with boxes that output 1080.
Special USB cards?
Cycling power?
Could you imagine Tivo HD owners going through this nonsense? I do it cause' I'm a glutton for tech geek. But for a list price of $250, none of these headaches should EVER happen.
Just my .02 cents.

[tomfisk]

Originally Posted by tvmaster2

It's a buggy bit of hardware, sadly. Mine locks up even without Sage - just open the Arcsoft Theater software and open the record window (and you don't have to hit "capture") - eventually it will crash. So that gets Sage off the hook.
What's the point of NOT using the spdif audio in? Why would I want to record HD video and not 5.1 audio?
Same goes for 1080 - that's the HD spec, so this thing should be able to deal with boxes that output 1080.
Special USB cards?
Cycling power?
Could you imagine Tivo HD owners going through this nonsense? I do it cause' I'm a glutton for tech geek. But for a list price of $250, none of these headaches should EVER happen.
Just my .02 cents.

I'll add my .02 cents. From my testing, yes HD-PVR hardware may not be perfect, but my testing has shown that having a "clean" HD signal coming from the STB as the root cause for the HD-PVR to lock. If the HD-PVR sucks up a "bad" HD signal, it will go off to neverland (not Michael's place!). I've done a simple fix of tuning to an SD channel, wait 10 seconds, then to an HD channel when my system comes out of standby (Moto DCH-6200). Knock on wood...I've recorded about 70 HD programs in the last 3+ weeks without a lockup.

[sleonard]

Originally Posted by tomfisk

I'll add my .02 cents. From my testing, yes HD-PVR hardware may not be perfect, but my testing has shown that having a "clean" HD signal coming from the STB as the root cause for the HD-PVR to lock. If the HD-PVR sucks up a "bad" HD signal, it will go off to neverland (not Michael's place!). I've done a simple fix of tuning to an SD channel, wait 10 seconds, then to an HD channel when my system comes out of standby (Moto DCH-6200). Knock on wood...I've recorded about 70 HD programs in the last 3+ weeks without a lockup.

This matches my experience. Of all the tips and tricks out there only 2 have had a noticable effect on my HDPVR reliability. The first was when I built a new Sage server with quality USB ports. This took me from 2-3+ lockups a day to 2-3 a week. The second was when Cox replaced my SA 3250HD boxes with SA4240HDC boxes and I haven't had a lockup since.

So, short story is quality hi-speed USB 2 ports and a quality signal from the source STB.

S

[jerryt]

Tool and instructions to place the HD-PVR on it's own USB controller;
http://1geek1tool.com/?s=usb+controller

[ewelin]

and that just confirmed that I in fact did have each HD-PVR on it's own USB controller.... I had high hopes for a few minutes though. LMAO.

[martin]

Originally Posted by ewelin

and that just confirmed that I in fact did have each HD-PVR on it's own USB controller.... I had high hopes for a few minutes though. LMAO.

I updated my list:

1. Do a clean install of the drivers
2. Revert to 1.0.5.301 drivers after using hwclear
3. Keep the HD-PVR cool
4. Don’t use SPDIF for audio, use RCA
5. Don’t use hardware versions earlier than E1
6. Defragment your hard drives
7. Lock component output to 720p
8. Use a NEC PCIe USB card
9. Filter power through a UPS
10. Increase Java memory
11. Isolate the HDPVR on its own physical controller

Has anyone tried watching the HD-PVR die with a USB sniffer? I'm wondering if a log file from the time it locks up would provide any clues?

@sleonard: how do I recognize a quality signal from the source STB? Once I have tuned an HD channel, does the signal change during commercials? Where I live you can own your STB, and I bought three Motorola DCT6200 boxes. Would it be worth a try to get a Pace Tahoe 775D? I went with the older Motorola set top boxes so I could change channels using firewire. I'm already getting divorced over this so there is nothing stopping me from throwing good money after bad.

[Chriscic]

It's great all the thinking that's going on here in trying to isolate the issue.

There have been many theories, but nothing yet has proven to be true, e.g. repeatable for other users.

Now that my system has gone to crap (after 6 months of flawless operation with two HD-PVRs) without me changing *anything at all*, I'm clueless and backing away form my prior conclusion that some units are just defective. Unless they somehow degrade over time, and I don't even want to go there.

I guess it's also conceivable that Dish did some sort of firmware "upgrade" themselves on my receivers overnight and something has changed there.

It may be worth noting that when I had no lock-ups I rarely had any audio-sync issues. Now I'm getting a lot, and my Survivor recording last week was off about 2-3 seconds by the end of the show.

I guess this is as good a time as any to move to the latest WHQL drivers, presumably reflash, and see what happens. I think I'm already running the latest drivers minus the WHQL certification though.

It just doesn't seem logical to me that the firmware would somehow degrade or get glitchy over time, and always manifest itself with the same types of lock-ups. It seems more likely the HD-PVR would bonk in all sorts of different ways with suddenly corrupted firmware.

[martin]

When a halt occurs on my system, Sage will keep attempting to restart the recording. The blue bling light on the HD-PVR repeatedly turns on for a few seconds, and then turns off, so that inidcates that USB is still working after a lockup. If I power off the server, HD-PVR, and set top boxes, unplug the USB cable, then plug it back in to a different port, the HD-PVR will recover and start recording to a new *.ts file after booting.

Originally Posted by Chriscic

It just doesn't seem logical to me that the firmware would somehow degrade or get glitchy over time, and always manifest itself with the same types of lock-ups. It seems more likely the HD-PVR would bonk in all sorts of different ways with suddenly corrupted firmware.

We can't assume our system will be stable if the Sage configuration hasn't changed. Storage space can fill up or get fragmented. Cable companies can push firmware updates or change signals that are sent to HD channels. Java or Microsoft can send updates. We need to consider the system as a whole rather than focusing on one thing. Once we can reproduce the bug, we can start a trial and error process to diagnose the source. Waiting for lockups to occur is too slow. We need to be able to create a lockup on demand.

[jerryt]

Originally Posted by martin

Once we can reproduce the bug, we can start a trial and error process to diagnose the source. Waiting for lockups to occur is too slow. We need to be able to create a lockup on demand.

I agree, but have not been able to reproduce lockup.

There are a few people here who can reproduce lockups by either opening the Arcsoft application and viewing video or by recording multiple channels and accessing the sage GUI.

So maybe the people who can reproduce these lockup could write a detailed process on duplicating the lockups. Then other people can help with the troubleshooting, like me...

By the way by my HD-PVR is on it's own enhanced USB controller !!!

[jerryt]

Originally Posted by martin

I updated my list:

1. Do a clean install of the drivers
2. Revert to 1.0.5.301 drivers after using hwclear
3. Keep the HD-PVR cool
4. Don’t use SPDIF for audio, use RCA
5. Don’t use hardware versions earlier than E1
6. Defragment your hard drives
7. Lock component output to 720p
8. Use a NEC PCIe USB card
9. Filter power through a UPS
10. Increase Java memory
11. Isolate the HDPVR on its own physical controller

I would just keep it simple;

8. Isolate the HDPVR on its own Enhanced USB controller (http://1geek1tool.com/?s=usb+controller), or Use a NEC PCIe USB card.

They are both methods for a dedicated USB controller.

[ewelin]

Originally Posted by jerryt

@ Ewelin
Were you able to get drivers 1.0.5.301 installed and working?

now running 1.0.5.301 as of 3 hours ago and it's working... Thanks a bunch jerryt!

[martin]

Originally Posted by jerryt

I would just keep it simple;

8. Isolate the HDPVR on its own Enhanced USB controller (http://1geek1tool.com/?s=usb+controller), or Use a NEC PCIe USB card.

They are both methods for a dedicated USB controller.

I kept those separate because I didn't want to assume they are the same problem. We don't know if the HD-PVR is compatible with only with NEC chips. I bought one of these cards last weekend and it didn't make any difference that I could see. I have never had a problem with USB compatibility, or errors like this. I have found devices like external storage get slower under a heavy load, but they never lock up. I noticed during one halt error using the NEC card that I locked both HD-PVR units. Curious.

[Chriscic]

I'm not buying the USB controller theory.

1) Rebooting the PC (without power cycling the HD-PVR) won't get the HD-PVR working again. You have to power cycle the HD-PVR.
2) Power-cycling the HD-PVR without rebooting *will* get things working again.

Some crazy interaction between the HD-PVR and USB controller = possible but not likely.

[jerryt]

Originally Posted by Chriscic

I'm not buying the USB controller theory.

1) Rebooting the PC (without power cycling the HD-PVR) won't get the HD-PVR working again. You have to power cycle the HD-PVR.
2) Power-cycling the HD-PVR without rebooting *will* get things working again.

Some crazy interaction between the HD-PVR and USB controller = possible but not likely.

Until someone can post a method of reproducing lockups, it will remain VooDoo...

[Chriscic]

Originally Posted by jerryt

Until someone can post a method of reproducing lockups, it will remain VooDoo...

Then someone please stop sticking pins into a doll of my WHS/Sage server.