HD300 very slow all of a sudden - Is it kaput?

[Flips]

My HD300 has been working great since I got it a few years ago. After returning from a 3 day vacation the HD300 was very slow all of a sudden. Navigating the UI would take a few seconds to go down to the next menu and then it would take another few seconds or so for the screen to update. Playing a video would be extremely choppy about a frame ever 3 seconds or so.

I switched to standalone mode and then I tried to play a recording (mpeg2) from the server. The video was very, very choppy, a few frames a second. Then I tried a MKV file and to my surprise it played just fine.

I tried firmware reload and factory reset from the service menu. I installed a new server on another computer, I moved to a new TV with a different network connection. All of my efforts haven't made a difference.

I am hoping that this isn't the end of the road for my HD300, but I am not sure there are many other things I can try. I am hoping some of you out there have some great suggestions for me.

Thanks,
flips

[sacrament055]

Did you try pulling the power and plugging it back in?

[Flips]

Originally Posted by sacrament055

Did you try pulling the power and plugging it back in?

Oh yeah.. many times..

[routerunner]

Originally Posted by Flips

My HD300 has been working great since I got it a few years ago. After returning from a 3 day vacation the HD300 was very slow all of a sudden. Navigating the UI would take a few seconds to go down to the next menu and then it would take another few seconds or so for the screen to update. Playing a video would be extremely choppy about a frame ever 3 seconds or so.

He's misbehaving because you left him alone, in the dark...you've should bring him on vacation with you , sorry I couldn't resist...

Seriously though...do you have another extender to try playing the same video? Can you isolate as much as you can the network, for instance a spare network switch between your server and the extender (removing your router out of the equation)? Or even better a direct cross network cable between the server and your extender? You'll be surprised on how network switches can start to go bad all of the sudden.

If you feel a little bit adventurous I can guide you on logging in into the unit and checks a couple of things like the memory usage or the current CPU usage, or if you want you can PM your IP address and I'll be happy to log into your unit to do some investigation...let me know

Eddy

[Flips]

Originally Posted by routerunner

He's misbehaving because you left him alone, in the dark...you've should bring him on vacation with you , sorry I couldn't resist...

You're right... maybe I should check the logs...

wow... okay... so I just fired it up tonight so I could telnet into it and see if anything looked off, per your suggestion, and it is working absolutely fine. Wow. that was the strangest thing... Now I wish I didn't spend a few hours today researching what my new client build was going to be..

Hopefully it stays this way... If this happens again I will try your network troubleshooting suggestion. Yesterday I did do a long series of pings between the server and the HD300 and they were all less than 1ms.

Here is what my client would have been:
$70 G620
$120 SRock Z68M-ITX/HT LGA1155 Mini-ITX
$50 Apex MI-008 Tower Black P4 Chassis with 250W Itx Psu+sata Power Supply
$30 4gb (2x2GB)
$60 32GB SSD
----
$330 Total

Thanks for your help,
flips

[routerunner]

Originally Posted by Flips

You're right... maybe I should check the logs...

wow... okay... so I just fired it up tonight so I could telnet into it and see if anything looked off, per your suggestion, and it is working absolutely fine. Wow. that was the strangest thing... Now I wish I didn't spend a few hours today researching what my new client build was going to be..

Hopefully it stays this way... If this happens again I will try your network troubleshooting suggestion. Yesterday I did do a long series of pings between the server and the HD300 and they were all less than 1ms.

Here is what my client would have been:
$70 G620
$120 SRock Z68M-ITX/HT LGA1155 Mini-ITX
$50 Apex MI-008 Tower Black P4 Chassis with 250W Itx Psu+sata Power Supply
$30 4gb (2x2GB)
$60 32GB SSD
----
$330 Total

Thanks for your help,
flips

No problem, one thing though...a couple of years ago you reported a problem within your network. I think you might want to run a wireshark to detect issues like conflicts or packet loss...

[Flips]

heh.. I forgot about that post...

Well I fired up the TV this morning and I was back to the same issue. It does seem to be a network issue. I plugged my laptop into that same cable and I was getting some pretty low transfer speeds on samba file transfers. I had ruled out network issues with the mkv file played, but that was a pretty small file, so the MB/S was low enough to not be an issue.

I put the sage server and client on a different switch and it worked fine. I tried to move everything over to that new switch and then everything stopped working... Then I gave up and enjoyed the nice weather..

I installed wireshark and captured some packets. Yikes is that a lot of information. I don't really know what I am looking at there, so I will have to play with that a bit.

At least I am resting comfortable that it isn't an HD300 issue.

flips

[routerunner]

Originally Posted by Flips

I installed wireshark and captured some packets. Yikes is that a lot of information. I don't really know what I am looking at there, so I will have to play with that a bit.

Yes, Wireshark can be a little daunting at first, however all the message entries are colour coded and what you should look at first are the red messages which implies packet retransmission and if I recall the yellow one which are ARP conflicts that in a LAN shouldn't happen really.

Also, the low transfer speed over a switch is normally a symptom of the port being very close to die, but that doesn't mean that the entire switch is faulty. Normally the ports in a switch are grouped into four that are driven by one Ethernet controller, so in a typical 16-ports switch you have two Ethernet controllers and so on. I had lots of this kind of failures in the past and it seems Linksys & Netgear the more exposed to that, but this is only speculation as mainly depends on the Ethernet chip being used than the switch brand itself.

[Flips]

I’ve got plenty of red… It looks like I am getting massive packet loss between the server and the HD300. I plug my laptop into the same connection and again I get a lot of packet loss on a smb transfer. Other connections I am pretty clean. I need to try to move the HD300 to another connection and look at wireshark tonight. Moving connections around on the switch didn’t seem to make much of a difference, but maybe I need to be more systematic about it.

Thanks again for the tip on wireshark. I played with Ethereal (what wireshark is built on apparently) years ago when I was in college, so this is bringing back some memories. I remember running the telnet filter on the capture and I was able to see every keystroke that my people on my floor were typing over a telnet session. That’s when I switched to ssh.

flips

[routerunner]

Originally Posted by Flips

I’ve got plenty of red…

I forgot...to be more specific there are different class of messages with a red background, in fact depending on the gravity, the foreground colour can be black, grey, green, yellow, but I don't recall which one is more serious...you need to play a little bit.

Wireshark is pretty good on highlighting network issues, and should be installed on every machine that run SageTV server as extenders are very susceptible to network fluctuations.

Eddy

[gdippel]

I suggest you check your hard disk(s). I had similar symptoms and it turned out that I had a flakey disk in a raid array. I replaced the disk and everything has been fine since then. I am still using the disk for non-critical applications so I suspect that streaming HD to my HD300 was just too demanding.

[Flips]

I played some more last night. On the same connection that I was getting little or no dropped packets on a smb transfer I was getting dropped packets on the HD300. When I have more time I am going to do a capture with just the server and the HD300 on its own switch.

Do you think it is possible that the HD300 is responsible for the dropped packets? The HD300's NIC is flaking out, maybe? I looked around for a usb wireless adapter, and it looks like there is only one model which is pretty hard to find now. Of course I missed the ebay sale.

I tried to ping the HD300 from the server with a large payload and I was getting a request timed out. I was able to get ~ 1ms responses from other devices on my network. Can someone try that on their HD300?
ping -t -l 65500 192.168.x.x.

For the dup ack errors they are black background with red letters which correlates to a TCP error. I was getting a lot more until I disabled CRC check offloading on the server nic.

I will check hard drives. Was your issue on the system drive or on the recording drive? I have the issue as soon as the home screen comes up and I try to navigate the menus. A couple of months ago I changed the system drive in the server to an SSD, so I am hoping there isn't an issue with that drive.

flips

[gdippel]

Originally Posted by Flips

I will check hard drives. Was your issue on the system drive or on the recording drive? I have the issue as soon as the home screen comes up and I try to navigate the menus. A couple of months ago I changed the system drive in the server to an SSD, so I am hoping there isn't an issue with that drive.
flips

It was the recording drive.

[routerunner]

Originally Posted by Flips

Do you think it is possible that the HD300 is responsible for the dropped packets? The HD300's NIC is flaking out, maybe?

My HD300 reports 24 errors since this morning at 7:30am when it gets switched on, however none of my HD200 reports errors and are all switching on at the same time.

I tried to ping the HD300 from the server with a large payload and I was getting a request timed out. I was able to get ~ 1ms responses from other devices on my network. Can someone try that on their HD300?
ping -t -l 65500 192.168.x.x.

With that payload is kind of expected on internet connection and rarely on intranet connections. I just tried over a VPN channel and it behaves like yours, however on the intranet works as expected, however I do have some timeout.

I was getting a lot more until I disabled CRC check offloading on the server nic.

Which is you server's NIC brand?

[Flips]

This was all on the intranet. I was getting every ping to time out, but I didn’t let it run for more than 5-10.

How do you count up the errors in wireshark? I wasn’t able to find that.

I am just using the mobo nic in the server, Atheros I think. I have a server grade intel nic laying around that I could throw in to see if that makes a difference.

----

Edit and OT: I just realized that you are the squeezeslave guy! That’s awesome work that you have going. I see that it is now a plugin. I will definitely have to check this out.

After seeing your post on squeezeslave I installed that on some of my PC clients and I one step closer to whole house audio. Now I am really glad I “cheaped out” and picked squeezebox instead of sonos for our kitchen player.

flips

[routerunner]

Originally Posted by Flips

How do you count up the errors in wireshark? I wasn’t able to find that.

Log into the unit and type "ifconfig", one of the information returned is the error count.

I am just using the mobo nic in the server, Atheros I think. I have a server grade intel nic laying around that I could throw in to see if that makes a difference.

I'll definitely try the Intel NIC. All my PCs at home have Intel NIC, every time I select a new mobo I'm very careful to get only server grade Intel NICs, never got it wrong.

Eddy

[routerunner]

Originally Posted by Flips

Edit and OT: I just realized that you are the squeezeslave guy!

yes, but don't tell anyone, it's a secret

[tmiranda]

Originally Posted by routerunner

yes, but don't tell anyone, it's a secret

It's a poorly kept secret!

[Flips]

The errors in ifconfig don't seem to be representative of what wireshark sees. Let it run overnight in standby which was showing photos all night and after 3.8 gigs of data I only had 3 errors, but I'm sure wireshark would have reported hundreds of TCP errors.

Anything look out of the ordinary here?

eth0      Link encap:Ethernet  HWaddr 00:XX
          inet addr:192.168.X  Bcast:192.168.X  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3064755 errors:3 dropped:0 overruns:0 frame:3
          TX packets:2935252 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:4120979518 (3.8 GiB)  TX bytes:224058109 (213.6 MiB)
          Interrupt:46

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2057 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2057 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1063945 (1.0 MiB)  TX bytes:1063945 (1.0 MiB)

flips

[drewg]

Originally Posted by Flips

For the dup ack errors they are black background with red letters which correlates to a TCP error. I was getting a lot more until I disabled CRC check offloading on the server nic.

I do 10GbE network drivers for a living, so I have to chime in here with a bit of clarification:

The "red" packets are any sort of TCP "error" (duplicate acks, bad checksums, etc). Almost all NICs do checksum offload these days, so the OS sends the packets to the NIC *without* calculating the checksum. Wireshark plugs in between the OS & NIC driver, so wireshark sees outgoing packets which have not had yet checksums calculated & falsely claims that their checksums are incorrect. By disabling the "CRC check offloading on the server nic", you're really just eliminating these false-positives. It does make the wireshark output easier to read, but (unless your server NIC's checksum offload is broken), it should not really change anything.

Now, if you want to find bad checksums on the HD300, do this:


cat /proc/net/snmp

For TCP, bad segments (TCP packets with bad checksums, or other fatal issues) is the 2nd to last item (InErrs). It is 0 on mine:

tango3[~]# cat /proc/net/snmp  | grep Tcp 
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts
Tcp: 1 200 120000 -1 146 4 9 0 3 81956055 39107458 35 0 11

Another interesting field is IP:

tango3[~]# cat /proc/net/snmp  | grep Ip 
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 2 64 81969349 0 1911 0 0 0 81967437 39107725 0 0 0 0 0 0 0 0 0

In this one, the one to watch is InHdrErrors (which is 0 above).


Drew