DHCP failures on MVP

lotus · #1 10-18-2006, 03:28 AM

I have been having issues with my MVPs not booting and fortunately have managed to get them to boot at last. In the end it seems my problem was caused by the same issue that many others have faced. My MVP was not obtaining an IP address from the DHCP server reliably.

When I observe the boot process in detail I see is that the MVP only waits a short time for DHCP server to reply and does not seem to have implemented the DHCP retry strategy specified in the DHCP RFCs. The retry strategy uses subsequent requests which are sent to the server at increasing intervals of up to 2 minutes when attempting to get an address until one is obtained.

If correct this may explain a lot of issues with the MVP and booting. Without DHCP retry any small issue causing the request packet to be lost or a delay in the DHCP response will cause the MVP to fail to boot. This may happen mysteriously as many other postings have stated as it is normal to have some packet loss in the network (particularly with wireless). It also happens when the enable times of network devices such as switches and server ports are longer than a couple of seconds as the DHCP request is sent and lost before the LAN port is up.

My MVP makes a DHCP request and gives up if the response has not come within a couple of seconds which makes the whole process unreliable. I’m new to this hardware and I see from other postings that others have studied this in detail. Can someone confirm that this is the behavior with their MVP (I only have version H3 units) and that DHCP requests are not retried until successful?

MrApollinax · #2 10-18-2006, 05:38 AM

Where excatly do you see the DHCP failure? Before or after the MVP.bin load? There are two DHCP requests made by the MVP. 1st is actually a BOOTP req for the MVP to boot off of a network image (MVP.bin). That req is acked by my SageTV server. The image then loads and mounts and then the MVP sends out a DHCP req on eth0. That req is acked by my router DHCP server. In either case I've seen my MVP send out reqs multiple times for each state.

lotus · #3 10-19-2006, 01:58 AM

Quote:

Originally Posted by MrApollinax

Where excatly do you see the DHCP failure? Before or after the MVP.bin load? There are two DHCP requests made by the MVP. 1st is actually a BOOTP req for the MVP to boot off of a network image (MVP.bin). That req is acked by my SageTV server. The image then loads and mounts and then the MVP sends out a DHCP req on eth0. That req is acked by my router DHCP server. In either case I've seen my MVP send out reqs multiple times for each state.

I can get the failure to occur in either of the DHCP requests in the boot process although the second one is more likely to fail. By failure, I mean that the MVP has proceeded past the DHCP stage of the boot process without obtaining an IP address.

If the failure occurs in the first part of the boot process the MVP makes a DHCP request and after a few seconds displays a message saying that an IP address was not obtained and proceeds to the scanning for servers screen. In the second part of the boot after the mvp.bin is loaded the MVP sends an address of 0:0:0:0 to the server (recorded in the log) shortly afterr starting to run.

I can also successfully boot if I ensure the DHCP responses come quickly. However in a practical network this is not going to be reliable. There are several threads in the forum that start out with an issue like "I woke up one day an found my MVPs no longer boot although they worked fine before and I haven't changed anything". Some change in the network load, DHCP server response delay, increased packet loss (particularly with woreless) will result in the DHCP failing on the MVP.

If the DHCP is well implemented (complies with the RFC) I would expect DHCP retries from the MVP to continue for more than a minute or preferably indefinitely until an address is obtained. What is the logic of proceeding with the next stage in the boot process without a valid IP address when nothing can work without one? To make matters worse once the DHCP fails the MVP can not automatically recover and requires a power cycle to get it to try again. Any suggestions on explaining this behavior to the technically savvy WAF?

THE POOR DHCP IMPLEMENTATION MAKES THE MVP VERY UNRELIABLE. For expample, if I connect the unit directly to my server LAN port on one interface it obtains IP addresses and boots fine but a second interface using different hardware fails due to a slighly longer time to enable the server port on that interface when the MVP enables its LAN port. I can connect a PC to either interface and obtain an IP address with no issue as they retry the DHCP as expected if the first request(s) come before the port is up. Other forum users report issues getting the MVP to boot with some brands of routers, switches and wireless devices etc that don't work with the MVP but work with other equipment. If their MVP behaves as mine does the issue is not with these devices but with the MVP DHCP implementation.

Nice to see your units send multiple DHCP requests. When you see the multilpe requests come from your MVP does it happen over a long period (minutes) of just a couple of seconds before the unit gives up?

MrApollinax · #4 10-19-2006, 05:56 AM

Quote:

Originally Posted by MrApollinax

Where excatly do you see the DHCP failure? Before or after the MVP.bin load? There are two DHCP requests made by the MVP. 1st is actually a BOOTP req for the MVP to boot off of a network image (MVP.bin). That req is acked by my SageTV server. The image then loads and mounts and then the MVP sends out a DHCP req on eth0. That req is acked by my router DHCP server. In either case I've seen my MVP send out reqs multiple times for each state.

Please note the bolded portion above in my statement. My MVP does send out multiple REQs in each case. I've seen this by doing packet captures on my network.

Quote:

In the second part of the boot after the mvp.bin is loaded the MVP sends an address of 0:0:0:0 to the server (recorded in the log) shortly afterr starting to run.

When you send out a DHCP discover packet you actually send out a broadcast from 0.0.0.0 to 255.255.255.255. If you dig deeper into the packet the dst MAC is ff:ff:ff:ff:ff:ff. This discover packet contains the info that you are looking to get from the DHCP server. The DHCP server should then respond in kind with a DHCP offer ACK.

In any case have you actually submitted a bug to SageTV. This is first and foremost a user forum where we try to help each other out. If you feel you have a legitamate bug you should submit a bug report.

lotus · #5 10-19-2006, 10:23 AM

Quote:

Originally Posted by MrApollinax

In any case have you actually submitted a bug to SageTV. This is first and foremost a user forum where we try to help each other out. If you feel you have a legitamate bug you should submit a bug report.

My reason for raising this on the forum is simple. I like the MVP when it works and I would like to use it with Sage. However I don't like to know that one day all my MVPs may not boot because of some component change in the network, DHCP server response or minor packet loss that I may not be able to identify easily or correct. Wireless LAN with this DHCP behavior would make failure very likely. Would definitely like Sage to have a robust DHCP implementation in the MVP especially before I let the WAF use them and was hoping to see that some more experienced users had identified and reported this issue to support already and/or a fix already exists and I really don't have a problem.

I have included the details on a support request I have previously opened when my MVP's would not boot before I identified the poorly implemented DHCP as the likely cause. I think other users may benefit from the input when investigating their MVP boot issues or deciding if the MVP is right (robust enough) for their situation. I certainly would have liked this information before I decided on the MVP instead of PC clients.

Sorry I wasn't more clear with the 0:0:0:0 address Sage reports when booting fails. Sage reports this in the log as the address coming from the MVP when DHCP fails. It reports the correct IP in the same point when DHCP works. I have no doubt your MVP sends out multiple DHCP requests. I also have packet traces of my MVP working and not working and understand the DHCP protocol well but am still learning about the MVP. Even without a packet trace its easy to see that proceeding on with the boot after a couple of seconds of DHCP attempts without an IP address isn't going to work well.

Would still like to know over what time period your MVP sends DHCP requests out in the hope of getting an IP as it could be related to the MVP firmware version or the particular mvp.bin used? This may help support investigate and solve this issue faster.

MrApollinax · #6 10-19-2006, 11:58 AM

When I did my testing I believe that I did my packet captures for 5 minutes. I haven't done any captures with v6, all my captures were don with v4 and v5 SageTV.

MrApollinax · #7 10-19-2006, 12:18 PM

Quick question, you said that the MVP timeout before your port comes up. Are you using a Cisco device? If so is portfast enabled for the switchport connected to the MVP? There were some issues with that in the past which can be found on this thread.

stevech · #8 10-19-2006, 01:27 PM

can the mvp be given a static IP and thus not do DHCP?

lotus · #9 10-19-2006, 10:04 PM

Quote:

Originally Posted by MrApollinax

Quick question, you said that the MVP timeout before your port comes up. Are you using a Cisco device? If so is portfast enabled for the switchport connected to the MVP? There were some issues with that in the past which can be found on this thread.

The port timeout is seen in my previous tests with a directly connected MVP. When the MVP disables and enables its LAN port the LAN interface on the server has to recover. The time taken by one of my ports is too long and one is OK (different hardware). When I use my simple gigabit switch at home and tftp32 as the DHCP server the MVP can boot.

This morning I took the MVP to work where we have some 11,000 devices mostly using DHCP on a campus LAN using high end Cisco infrastructure. MVP does not get an IP in this environment either. If I mention tweaking many things responsible for the delay to get DHCP to work to the network team I sure they will tell me to go away and fix my device. The higher end your infrasturcture the more features that can add delay (port security, vlan, DHCP relay, load from other users etc).

Appreciate your suggestion to work around the problem by shaving some time off the network devices to try to get the MVP DHCP to operate. Have already taken this approach to get it to work. If Windows had this DHCP behavior a good percentage of the worlds 100 million+ PCs would not have network access.

Edit ----
Received an update from support that this issue will be corrected in the upcoming beta 6.0.13

MrApollinax · #10 10-20-2006, 05:52 AM

Looks like you have a solution for your problem. Like I said, I've only done detailed catpures with v4 and v5. I have not installed v6 yet. It seems to have been a v6 problem?

steingra · #11 10-20-2006, 08:30 AM

I had wished for a static IP address on the MVP's too. Would seem to make any DHCP problems go away. I have noticed (occasionally) that the MVP's dont ever get an IP address. I just never had time to look into it in detail. I havent officially rolled out all the MVP's in the house yet. So the WAF isnt an issue *yet*.

So I guess if version 6 of SageTV is going to fix this issue, then static IP isnt a huge deal. But sometimes, I always like to know the IP address of every device. And at least if I assign them all by hand I know what's what on the network. Without making things overly complicated

mdnttoker · #12 10-20-2006, 11:01 AM

I don't really understand how Sage has anything to do with static IP addresses.

This is an issue that Hauppauge would have to fix in the hardware/software on the MVP. And given their receptiveness in the past, I'd say the chances of getting a static IP are zilch and none.

-Dan

stevech · #13 10-20-2006, 12:27 PM

Quote:

Originally Posted by steingra

So I guess if version 6 of SageTV is going to fix this issue, then static IP isnt a huge deal.

Why is a DHCP protocol exchange a Sage problem? Is it that Sage doesn't close a stale TCP connection quickly enough?

JParedis · #14 10-20-2006, 02:39 PM

Quote:

Originally Posted by mdnttoker

I don't really understand how Sage has anything to do with static IP addresses.

This is an issue that Hauppauge would have to fix in the hardware/software on the MVP. And given their receptiveness in the past, I'd say the chances of getting a static IP are zilch and none.

-Dan

Agree. I'm afraid it's build-in in the base software of the MVP, so it can only be fixed in new" hardware" version (haven't seen a flash update option for MVP's

).

I noticed that my MVP did take more than 5 minutes to start, the sagetv log file showed the MVP did try to make connection but was refused as it had no IP (the Hauppauge service I believe is capable to give than an IP). After a while it came suddenly with an IP and all started working. The second 'reboot' of the MVP, than with the SageTV software, had no issues. However, that may be because the DHCP still had the MAC/IP in its client table and was quicker in assigning it (again). Not sure about this.

I'm going to do some network logging (all new to me, so learning on the way). I did notice in my first one that the boot of the MVP is slightly different than the one of the SageTV soft on it. But again, that was my first time I used Ethereal - so going to do it again. FYI:

=> MVP boot (numbers are actual line numbers in log)
1. MVP (0.0.0.0) broadcast - DHCP - DHCP Discover
2. DHCP broadcast - ARP - Who has IP.100
3. DHCP broadcast - DHCP - DHCP Offer
4. MVP (IP.100) broadcast - UDP - (looking for service)
5. HTPC (IP.101) broadcast (IP.255) - UDP - (announcing service availability)
6. MVP (IP.100) broadcast - ARP - Who has IP.101 (=htpc)
7. HTPC MVP - ARP - reply
than communication starts.

=> MVP second boot (SageTV software)
12333. MVP (0.0.0.0) broadcast - DHCP - DHCP Discover
12334. DHCP broadcast - ARP - Who has IP.100
12335. DHCP broadcast - DHCP - DHCP Offer
12336. MVP (0.0.0.0) broadcast - DHCP - DHCP Request
12337. DHCP broadcast - DHCP - DHCP ACK
12338. MVP (IP.100) broadcast - UDP - (looking for service)
12339. HTPC (IP.101) broadcast (IP.255) - UDP - (announcing service availability)
12340. MVP (IP.100) broadcast - ARP - Who has IP.101 (=htpc)
12341. HTPC MVP - ARP - reply
Than some SYN ACK's and communication starts.

I do not (yet) understand the difference between the DHCP process of Hauppauge's MVP and the SageTV MVP soft.

Some study to do.

mdnttoker · #15 10-20-2006, 03:24 PM

I've been conversing with SageTV support for awhile regarding this issue.

(BTW: Wireshark is the new Ethereal, if you want the latest versions).

I'm a bit of a noob on this networking stuff too, so I haven't really tried to decode the stuff myself yet. I just send the pcap files onto Sage Support.

It's been over a month of back and forth with Sage without a fix though, so I may start doing a little investigating myself as I get more impatient...

-Dan

prushing · #16 10-25-2006, 12:19 PM

What has been described here is very similar to the behavior I am having, except that Sage tell me the MVP I am using is a pre-production evaluation unit (based on the log file I sent and the MAC address of my MVP) and Hauppauge tells me my unit is fine and that Sage is smoking crack, essentially.

The Haupapuge SW boots it every time, no problem, Sage will not. I get the connecting screen at best.

I've tried every MVP.bin I could get my hands on.

I bought the device from a Radioshack outlet in for $35 bucks. It would have been great if it worked, alas, I have other ones the play around with and have a backup remote if i break one of my other ones...

Anybody else haev problems with a MVP that works fine everywhere else, except with Sage?

mdnttoker · #17 10-25-2006, 01:02 PM

Quote:

Originally Posted by prushing

What has been described here is very similar to the behavior I am having, except that Sage tell me the MVP I am using is a pre-production evaluation unit (based on the log file I sent and the MAC address of my MVP) and Hauppauge tells me my unit is fine and that Sage is smoking crack, essentially.

The Haupapuge SW boots it every time, no problem, Sage will not. I get the connecting screen at best.

I've tried every MVP.bin I could get my hands on.

I bought the device from a Radioshack outlet in for $35 bucks. It would have been great if it worked, alas, I have other ones the play around with and have a backup remote if i break one of my other ones...

Anybody else haev problems with a MVP that works fine everywhere else, except with Sage?

Yup, right here.

GB-PVR and Hauppauge software both load fine across my Powerline network adapters.

Sage does not (infinite "Loading Application" screen). However, if I do an initial load of Sage locally via a switch and a local laptop to the MVP, the Sage MVP software loads up fine and streams everything I ask it to from my server across the Powerline adapters.

In my last support request, Sage told me I needed to update the firmware on the MVP to get it to do the initial load across the powerline adapters. I contacted Hauppauge and was told it is impossible to update the MVP firmware (only the run-time software can be updated).

Seems Sage has a ways to go on building a robust TFTP server that equals that provided by Hauppauge and other software packages...

Prushing: what is your network topology? And have you tried the most recent GB-PVR build? (98.13?)

stevech · #18 10-25-2006, 01:33 PM

Quote:

Originally Posted by prushing

The Haupapuge SW boots it every time, no problem, Sage will not. I get the connecting screen at best.

Is Sage the bootp server for the MVP? If so, we can stop the blame-game and state that it's Sage at fault.

lotus · #19 10-26-2006, 03:39 AM

Quote:

Originally Posted by stevech

Is Sage the bootp server for the MVP? If so, we can stop the blame-game and state that it's Sage at fault.

It seems both Hauppauge and Sage are at fault!

I have received replies from Sage support confirming they will be improving the DHCP in the second phase of the boot process (the part controlled by the mvp.bin) in the next beta. In my tests the bin files provided by others (Hauppauge, GBPVR) do work much better than the mvp.bin from Sage. Postings I see on other forums from users having MVP problems seem to be mostly from the first DHCP request failing because of the lack of sufficient retries if an IP address is not received quickly.

The initial DHCP request from the firmware in the MVP is made from the Hauppauge code in the MVP before the transfer of the bin file. This is the same regardless of who is providing the bin file. It does not work better or worse with Sage, GBPVR or the standard Hauppauge software. Sage support reports that they are working with Hauppauge to resolve the issue with the first DHCP request. No indication of how long to provide a fix.

hemicuda · #20 10-28-2006, 09:43 AM

There's another thread on this issue too if you haven't seen it yet.

http://forums.sagetv.com/forums/showthread.php?t=20248

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)