Linux 64-bit HDHR Capture Fails

[Taddeusz]

Ok, I've been doing some testing of this. Everything seems to work fine on 32-bit. It's just on 64-bit that capture from an HDHR doesn't work. I enabled debug logging on both 32-bit and 64-bit to compare the two logs to get an idea where I need to look in the code to troubleshoot.

Here are the two logs I've captured this morning. I can see where the 32-bit machine successfully starts capturing video and that the 64-bit machine fails and eventually says that the signal was lost. I still don't have a clue where to start.

Looking at the code my first thought is that the 64-bit Java pointer conversion in native/so/HDHomeRun2.0/sage_HDHomeRun.cp is somehow the problem on 64-bit but gets handled correctly on 32-bit. I am likely wrong though.

[mwnswiss]

What model of HDHR do you have and what broadcast standard are you using. With my HDHR DUAL EU I never get so far with the built in library. I have more success with the dvbhdhomerun kernel mod. Wondering if the models have an impact or if there is somewhere a config which is different and blocks my HDHR device to work. (I have as well newest firmware on my HDHR DUAL).

[Taddeusz]

Originally Posted by mwnswiss

What model of HDHR do you have and what broadcast standard are you using. With my HDHR DUAL EU I never get so far with the built in library. I have more success with the dvbhdhomerun kernel mod. Wondering if the models have an impact or if there is somewhere a config which is different and blocks my HDHR device to work. (I have as well newest firmware on my HDHR DUAL).

I'm in the United States so my HDHR's are capturing over-the-air ATSC or cable QAM. We don't have access to DVB here. I have 3 HDHR's. Two of them are the older beige dual tuner model. One is ATSC and the other is QAM. The third is a newer black dual tuner HDHR2 is also ATSC. Have not tried testing between the different models. Also have not yet tested QAM. I've only been testing from my my older ATSC connected tuner.

The HDHR I have hooked to cable at the moment is actually not in regular use. It is a spare I had laying around because one of the two tuners doesn't get a good signal. I hooked it back up because I was having trouble picking up channels on one of my TV's. Turned out my cable company moved channels around on us.

[Taddeusz]

I will still test but I doubt there will be a difference between different models or broadcast standards. As far as the HDHR's are concerned the only difference there should be is in the way tuners tune and decode the signal to get the video. Assuming you're using the built-in HDHR support the transfer of the video from the HDHR to SageTV should be identical in any case.

[coppit]

I know nothing about this stuff, but it looks to me like what's happening is that SageTV has something that streams the video directly to disk. Once the VideoFrame thread sees that there's data in that file, it then starts the MiniPlayer on that file for the client.

VF waiting for data to appear in new file...liveWait=1789
VF thread is now waiting for 0:00:01.789
VF processing job VFJob[LoadMF r=0.0 t=0 file=MediaFile[id=82178 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest32 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 3576 kbps [#0 Video[MPEG2-Video 29.97003 fps 1920x1080 16:9 interlaced id=e0]#1 Audio[AC3 48000 Hz 6 channels 384 kbps MAIN idx=1 id=bd-80010000 eng]#2 Audio[AC3 48000 Hz 2 channels 192 kbps idx=2 id=bd-81010000 spa]] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:35:39.359-Wed 12/31 18:00:00.000]] ifn=null] nPlayin=false
VF network encoder playback detected: false
VF processing on UI Thread VFJob[LoadMF r=0.0 t=0 file=MediaFile[id=82178 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest32 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 3576 kbps [#0 Video[MPEG2-Video 29.97003 fps 1920x1080 16:9 interlaced id=e0]#1 Audio[AC3 48000 Hz 6 channels 384 kbps MAIN idx=1 id=bd-80010000 eng]#2 Audio[AC3 48000 Hz 2 channels 192 kbps idx=2 id=bd-81010000 spa]] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:35:39.359-Wed 12/31 18:00:00.000]] ifn=null]
VideoFrame creating new media player for file:MediaFile[id=82178 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest32 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 3576 kbps [#0 Video[MPEG2-Video 29.97003 fps 1920x1080 16:9 interlaced id=e0]#1 Audio[AC3 48000 Hz 6 channels 384 kbps MAIN idx=1 id=bd-80010000 eng]#2 Audio[AC3 48000 Hz 2 channels 192 kbps idx=2 id=bd-81010000 spa]] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:35:39.359-Wed 12/31 18:00:00.000]]
VF file=MediaFile[id=82178 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest32 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 3576 kbps [#0 Video[MPEG2-Video 29.97003 fps 1920x1080 16:9 interlaced id=e0]#1 Audio[AC3 48000 Hz 6 channels 384 kbps MAIN idx=1 id=bd-80010000 eng]#2 Audio[AC3 48000 Hz 2 channels 192 kbps idx=2 id=bd-81010000 spa]] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:35:39.359-Wed 12/31 18:00:00.000]] targetTime = Thu 8/20 8:35:39.359
VideoFrame.timeSelected(Thu 8/20 8:35:39.359, true) currFile=MediaFile[id=82178 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest32 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 3576 kbps [#0 Video[MPEG2-Video 29.97003 fps 1920x1080 16:9 interlaced id=e0]#1 Audio[AC3 48000 Hz 6 channels 384 kbps MAIN idx=1 id=bd-80010000 eng]#2 Audio[AC3 48000 Hz 2 channels 192 kbps idx=2 id=bd-81010000 spa]] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:35:39.359-Wed 12/31 18:00:00.000]]
initDriver0()
Issuing MCSR client socket to 001cc0dac01b
MiniPlayer established for 001cc0dac01b

The problem is that it waits forever for data to hit the file:

VF waiting for data to appear in new file...liveWait=-303
VF thread is now waiting for 0:00:00.200
VF processing job VFJob[LoadMF r=0.0 t=0 file=MediaFile[id=82171 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest64 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 0 kbps [] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:26:14.853-Wed 12/31 18:00:00.000]] ifn=null] nPlayin=false
VF network encoder playback detected: false
VF waiting for data to appear in new file...liveWait=-504
VF thread is now waiting for 0:00:00.200
<repeats ad nauseum>

Backing up to where the recording is supposed to start:

[AsyncWatch] startEncoding for HDHomeRun 10193bc0 Tuner 1, file=/var/media/tv/GoodMorningAmerica-5159-0.mpg, chan=7-5-1
[AsyncWatch] HDHR_setInput0(0x7f15d84fc1e0, 100, 0, Air, 1, 1)
[AsyncWatch] HDHRCaptureDevice->doTuneChannel(7-5-1,true)
[AsyncWatch] HDHR_setChannel0(0x7f15d84fc1e0, 7-5-1, true, 31)
[FinalRender-001cc0dac01b] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/ProgressBar3.png width=500 height=29 ptr=373
[AsyncWatch] HDHR_setupEncoding0(0x7f15d84fc1e0, /var/media/tv/GoodMorningAmerica-5159-0.mpg, 0)
[AsyncWatch] HDHR_getBroadcastStandard0(0x7f15d84fc1e0)
[HDHomeRun 10193bc0 Tuner 1-Encoder] Starting capture thread for HDHomeRun 10193bc0 Tuner 1
[AsyncWatch] VF.submitJob(VFJob[WatchMF r=0.0 t=0 file=MediaFile[id=82171 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest64 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 0 kbps [] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:26:14.853-Wed 12/31 18:00:00.000]] ifn=null])
[VideoFrame-001cc0dac01b] VF processing job VFJob[WatchMF r=0.0 t=0 file=MediaFile[id=82171 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest64 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 0 kbps [] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:26:14.853-Wed 12/31 18:00:00.000]] ifn=null] nPlayin=false
[VideoFrame-001cc0dac01b] VF processing on UI Thread VFJob[WatchMF r=0.0 t=0 file=MediaFile[id=82171 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest64 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 0 kbps [] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:26:14.853-Wed 12/31 18:00:00.000]] ifn=null]
[VideoFrame-001cc0dac01b] VF processing job VFJob[LoadMF r=0.0 t=0 file=MediaFile[id=82171 A[5159,4195,"Good Morning America",32312@0820.07:00,120,T] mask=TV host=sagetest64 encodedBy=HDHomeRun 10193bc0 Tuner 1 KOCODT format=MPEG2-PS 0:00:00 0 kbps [] /var/media/tv/GoodMorningAmerica-5159-0.mpg, Seg0[Thu 8/20 8:26:14.853-Wed 12/31 18:00:00.000]] ifn=null] nPlayin=false

I suppose it could be something as simple as the channel not changing right.

[Taddeusz]

It could be a problem with capturing data from the HDHR or it could be a problem writing the data to the disk. Or somewhere in between those two events.

[Narflex]

So the VideoFrame is making a call into the MediaFile objects it's trying to play and checking MediaFile.shouldIWaitToStartPlayback(). This waits on 2 things, the file format being properly detected and there being enough data in the file to start playback. Generally, you won't have a file format if you don't have enough data to detect it (except for devices that have fixed formats like legacy analog capture devices).

So the first thing to check is if the file it's trying to write to is actually increasing in size, or if its just stuck at zero bytes...my guess is its stuck at zero bytes and nothing is getting written.

You can also put a file named 'NATIVE_LOG.ENABLE' in your SageTV directory, and then it should create a Native.log file which will log a bunch more data from the native capture layer which may help in identifying what's actually going wrong.

[Taddeusz]

Ok, here are the Native.log files for both 32-bit & 64-bit. Yes, you're right. The file size is stuck at 0 and never grows.

Is the parser initated by a file size change?

[Narflex]

I took a look at these and the data is coming in from the HDHR. The problem is in the stream parsing...in the 32 bit version it's finding the audio & video stream...but in the 64 bit version, it's only finding the audio stream.

32-bit:

08/24/2015 15:55:07 Program Filter(delayed): Set program ID:0x7 [71,74,0,0,0,0,0,0] (result = 0)
08/24/2015 15:55:07 ES:0:0 strmid:0xe0-0x0 esid:0x0000 video MPEG2 "open"
08/24/2015 15:55:07 ES:0:1 strmid:0xbd-0x0 esid:0x0000 private MPEG2
08/24/2015 15:55:07 AV:0:1 audio AC3 (pos:0)
08/24/2015 15:55:08 AV:0:0 video MPGV (pos:1681848)

64-bit:
08/24/2015 15:48:10 Program Filter(delayed): Set program ID:0x7 [71,74,0,0,0,0,0,0] (result = 0)
08/24/2015 15:48:11 ES:0:0 strmid:0xe0-0x0 esid:0x0000 video MPEG2 "open"
08/24/2015 15:48:11 ES:0:1 strmid:0xbd-0x0 esid:0x0000 private MPEG2
08/24/2015 15:48:11 AV:0:1 audio AC3 (pos:0)

Everything else seems identical, with the exception of when it's printing out the size of a structure that's being passed into the PMTDump method (which is 180 on 32 bit and 192 on 64 bit).

Let me see if Qian is back yet...he can figure this out way faster than me since he wrote that code.

[mwnswiss]

Can confirm the same on my Linux, 64bit, HDHR3-4DC DVB-C Europe, using native libhdhomerun (not the dvbhdhomerun kernel module)

I can record a radio station (audio only), this works fine, it does not work with video, zero size file.

[bcjenkins]

Have you tried the prime_encoder plugin from jpwegas? I think it forgoes the built in library and even allows for you to specify a different build of ffmpeg. It will also remove the two tuner limit on the built in library as well

[Taddeusz]

To give an update on this Qian is working with me on a solution to this issue. Currently video capture is working but the dimensions & frame rate are not being parsed properly resulting in the video looking weird in Placeshifter.

[jvl711]

I noticed while testing some changes with the PrimeNetEncoder on a Linux x64 build of SageTV that the files I was capturing from HDHomeRun Prime externally also were not getting the video portion of the file recognized. I am wondering if this is possibly a more systemic issue than just the HDHomeRun capture code.

Josh

[Taddeusz]

Originally Posted by jvl711

I noticed while testing some changes with the PrimeNetEncoder on a Linux x64 build of SageTV that the files I was capturing from HDHomeRun Prime externally also were not getting the video portion of the file recognized. I am wondering if this is possibly a more systemic issue than just the HDHomeRun capture code.

Josh

It is. Qian hasn't shared the actual code with me but the libraries he has sent me to test have been the Native Core & TScore libraries. That being said, libhdhomerun gets compiled into the NativeCore. As I said before he got the file output working. Now it's a matter of getting the parser to output the correct values.

The root cause of at least most of this has to do with the size difference of number types in C between 32-bit and 64-bit. The code is written to expect a certain size that correlates to the sizes of those number types in 32-bit. Making that code compatible with both 32-bit and 64-bit can be difficult.

Qian has limited time available to work on this so it's a slow process.

[Taddeusz]

I misspoke. libhdhomerun is part of libHDHomeRunCapture.so. Not libNativeCore.so.

[mwnswiss]

Any progress on the 64bit issue? I could test if I get some code.

[Taddeusz]

Originally Posted by mwnswiss

Any progress on the 64bit issue? I could test if I get some code.

Qian has it working. Not sure when the code will be submitted for a pull request though. May be waiting on further input for other issues.

[donkey]

I have a fork for one who wants to test https://github.com/qianzhang5/sagetv, which refactored NativeCore variable declaration for 32-bit 64-bit compatible.

Taddeusz and I tested code on 64-bit build, I really hope some one to pull, build and test on 32-bit machine before I submit a pull request to merge
So far there is one issue in 64-bit version build with my fix that hasn't been solved. After channel scanning, the empty channels are listed on a lineup. Jeff points out there may be a bug in STV code that doesn't drop out an empty channel (empty scan result string).

[mwnswiss]

Compiled your branch on Ubuntu 14.04 using a HDHR EXPAND 4DC with 4 DVB-C and it worked! Video is correctly saved and I can watch it.

Excellent work, it seems that any kind of plain ANSI C types can cause issues on interfaces covering return and argument variables, we need to make sure we use the correct types like uint32_t, correct?

I cannot test currently on 32bit, but would be good to get feedback from a linux 32bit test.

Originally Posted by donkey

....So far there is one issue in 64-bit version build with my fix that hasn't been solved. After channel scanning, the empty channels are listed on a lineup. Jeff points out there may be a bug in STV code that doesn't drop out an empty channel (empty scan result string).

I observed that as well, probably it is a bug in STV but I see another bug as well in the HDHomeRun library which deals with the channels. In my case using DVB-C or DVB-T in Europe, SageTV shows me all Radio stations and all empty stations on the OpenSource version using this native HDHomeRun. On the V7 using the dvbhdhomerung kernel module which emulates the hdhomerun to be a plain BDA device it works correctly, so I guess the channel handling in HDHomeRun is not complete and provides too much data to SageTV. I will do some further work on it.

[mwnswiss]

Could find some time to setup a Ubuntu 14.04 32bit system, could compile and run your branch, recorded video is fine.