PrimeNetEncoder - HDHomeRun Prime Network Encoder

[jvl711]

Originally Posted by EnterNoEscape

I guess hdhomerun_config.exe is setting the stream of UDP packets to your computer IP address. The fact that it's UDP is enough to make me think those occasional missed packets might be network issues or congestion. I have seen small glitches myself using SageDCT when moving large files to and from my primary SageTV server. I was thinking about creating a subnet just for recording devices so I can try to guarantee bandwidth for my tuners.

On the InfinitTV devices as far as I have been able to determine, you can't set the IP address for the stream via upnp. You have to either connect to rtsp://x.x.x.x:8554/cetonmpeg[0-5] (you retrieve the URL via an action on one of the services on one of the embedded devices) and tell it to stream to you and what port to stream to. The alternative is to set the IP address by using POST on the web interface which I'm avoiding since I want to do this as proper as possible. After looking at a few implementations of this protocol, I think I can just write my own. It doesn't appear to be that complicated.

As for the SageTV protocols; I was following the "blueprint" in EncodingServer.java. I want to implement auto discovery. At first I was a little puzzled with PROPERTIES. I think this was mostly because this is actually my first time programming in java (incredibly similar to C# and even easier to use in some regards). So as I understand it java.util.Properties can directly import the text in .properties files and it can export in either XML or in the linear format that SageTV uses by default. So all this is returning in the first line is the number of properties, then the next lines are literally the properties as they would appear in the .properties file. Honestly the comment in the code only helped to confuse me. Do I have that correct? Also I'm unclear on device_class and encoding_host. In the for loop, they are excluded as expected based on the comment, but I don't see where they are modified and added to the returned data. These properties are not inferred and do need to be set to "NetworkEncoder" and "iport" respectively correct?

I was going to just look at the code behind these other commands, but maybe you already know what they are looking for. I do want to implement channel scanning so I can do a scan that will actually detect CopyOnce channels and tell SageTV they don't have a signal. I think this could make initial setup easier for many people.
GET_START - Is this just the starting size of the recording?
AUTOTUNE - Channel scanning?
AUTOSCAN - Channel scanning?
AUTOINFOSCAN - What are normal return values for this command?

Thanks a bunch for the help! I feel like I'm on the right track here (I've only been working on this for about a week) despite being very ambitious. I really got my motivation back when SageTV was open sourced and I really want to contribute to the reason SageTV is so great.

Joe

Here is my method for sending the properties. This will probably be a little helpful. It is as simple as you said. Number of properties, then each property \r\n terminated.

private String getPropertiesMessage()
    {
        String props = "";
        
        props += "31\r\n";
        
        props += "mmc/encoders/31035317/100/0/brightness=-1\r\n";
        props += "mmc/encoders/31035317/100/0/broadcast_standard=\r\n";
        props += "mmc/encoders/31035317/100/0/contrast=-1\r\n";
        props += "mmc/encoders/31035317/100/0/device_name=" + this.getTunerName() + "\r\n";
        props += "mmc/encoders/31035317/100/0/encode_digital_tv_as_program_stream=false\r\n";
        props += "mmc/encoders/31035317/100/0/hue=-1\r\n";
        props += "mmc/encoders/31035317/100/0/saturation=-1\r\n";
        props += "mmc/encoders/31035317/100/0/sharpness=-1\r\n";
        props += "mmc/encoders/31035317/100/0/tuning_mode=Cable\r\n";
        props += "mmc/encoders/31035317/100/0/tuning_plugin=\r\n";
        props += "mmc/encoders/31035317/100/0/tuning_plugin_port=0\r\n";
        props += "mmc/encoders/31035317/100/0/video_crossbar_index=0\r\n";
        props += "mmc/encoders/31035317/100/0/video_crossbar_type=100\r\n";
        props += "mmc/encoders/31035317/audio_capture_device_name=\r\n";
        props += "mmc/encoders/31035317/capture_config=2000\r\n";
        props += "mmc/encoders/31035317/default_device_quality=Great\r\n";
        props += "mmc/encoders/31035317/delay_to_wait_after_tuning=0\r\n";
        props += "mmc/encoders/31035317/device_class=\r\n";
        props += "mmc/encoders/31035317/encoder_host=" + this.localIPAddress + ":" + this.port + "\r\n";
        props += "mmc/encoders/31035317/encoder_merit=0\r\n";
        props += "mmc/encoders/31035317/encoding_host=" + this.localIPAddress + ":" + this.port + "\r\n";
        props += "mmc/encoders/31035317/fast_network_encoder_switch=false\r\n";
        props += "mmc/encoders/31035317/forced_video_storage_path_prefix=\r\n";
        props += "mmc/encoders/31035317/last_cross_index=0\r\n";
        props += "mmc/encoders/31035317/last_cross_type=100\r\n";
        props += "mmc/encoders/31035317/live_audio_input=\r\n";
        props += "mmc/encoders/31035317/multicast_host=\r\n";
        props += "mmc/encoders/31035317/never_stop_encoding=false\r\n";
        props += "mmc/encoders/31035317/video_capture_device_name=" + this.getTunerName() + "\r\n";
        props += "mmc/encoders/31035317/video_capture_device_num=0\r\n";
        props += "mmc/encoders/31035317/video_encoding_params=Great\r\n";
        
        return props;
    }

I actually do not implement any of those commands. I only use 'VERSION', 'PROPERTIES', 'NOOP', 'START', 'STOP', 'GET_FILE_SIZE' and 'QUIT'. I am using version 3 of the protocol so that I get the uploadID. I do not think there is any way to tell the server you do not implement a command, so I just respond 'OK' for any unknown command. I am not sure if there is a better way to handle that.

[EnterNoEscape]

I see what I needed for PROPERTIES. Thank you!

[jvl711]

Originally Posted by EnterNoEscape

I see what I needed for PROPERTIES. Thank you!

No problem! I hope the info was helpful.

[EnterNoEscape]

So I wrote out my own communication methods for doing RTSP communication that are working very well. Now, I have started to do raw dumps from the RTP streams. Initially I was very confused because it seemed like it was missing several packets at random. I set up a thread-safe non-blocking queue so that the packets can be produced independent of the thread that processes them since I want to ultimately be able to ID the I frames when needed to implement the SWITCH command and I don't want processing overhead to potentially miss an incoming frame. Anyhow, the thing that fixed it was increasing the receiveBufferSize value. Apparently on Windows it defaults to 8192. I bumped it up several magnitude and it fixed the problem immediately. Maybe this parameter, if you are not already using it, could help with the occasional dropped packets you're seeing.

Also are you using the bytes at index 2-3 to determine that you are missing a packet or are you seeing that somewhere else?

[jvl711]

Originally Posted by EnterNoEscape

So I wrote out my own communication methods for doing RTSP communication that are working very well. Now, I have started to do raw dumps from the RTP streams. Initially I was very confused because it seemed like it was missing several packets at random. I set up a thread-safe non-blocking queue so that the packets can be produced independent of the thread that processes them since I want to ultimately be able to ID the I frames when needed to implement the SWITCH command and I don't want processing overhead to potentially miss an incoming frame. Anyhow, the thing that fixed it was increasing the receiveBufferSize value. Apparently on Windows it defaults to 8192. I bumped it up several magnitude and it fixed the problem immediately. Maybe this parameter, if you are not already using it, could help with the occasional dropped packets you're seeing.

Also are you using the bytes at index 2-3 to determine that you are missing a packet or are you seeing that somewhere else?

Sounds like you are making great progress! Yes I did increase the receiveBufferSize. That did help round off small processing delays. I am not identify the packets. I am tracking the length of time between receiving packets. I am seeing occasional long delays. 1500-2500 ms. I kept increasing the logging in my application to try and determine where this delay is coming from. I ruled out the following.

1. My network was dropping packets. (Confirmed that this is not a problem with tests with HDHomeRun_config app).
2. blocking wait writing to ffmpeg process or mediaserver (confirmed this was not an issue by monitoring the size of the buffered output streams)

I am now looking into potential garbage collection pauses. I am thinking that I create a ton of objects on the heap with all of the input output stream transfers that occur. There are byte arrays everywhere!!! I am trying a concurrent garbage collection profile to see if that fixes it.

[Fuzzy]

Originally Posted by jvl711

Sounds like you are making great progress! Yes I did increase the receiveBufferSize. That did help round off small processing delays. I am not identify the packets. I am tracking the length of time between receiving packets. I am seeing occasional long delays. 1500-2500 ms. I kept increasing the logging in my application to try and determine where this delay is coming from. I ruled out the following.

1. My network was dropping packets. (Confirmed that this is not a problem with tests with HDHomeRun_config app).
2. blocking wait writing to ffmpeg process or mediaserver (confirmed this was not an issue by monitoring the size of the buffered output streams)

I am now looking into potential garbage collection pauses. I am thinking that I create a ton of objects on the heap with all of the input output stream transfers that occur. There are byte arrays everywhere!!! I am trying a concurrent garbage collection profile to see if that fixes it.

Are you seeing the delays only when running in the sagetv process, or also when your encoder is also in it's own JVM? There is a LOT more overhead going on in the sagetv JVM that may be blocking you out occasionally.

[EnterNoEscape]

Originally Posted by jvl711

I am now looking into potential garbage collection pauses. I am thinking that I create a ton of objects on the heap with all of the input output stream transfers that occur. There are byte arrays everywhere!!! I am trying a concurrent garbage collection profile to see if that fixes it.

Perhaps you can subscribe to the garbage collection event and have it write out a log entry so you can attempt to correlate it. You brought something up that I had not considered might interfere with processing. I'll be keeping an eye out for that in my own implementation.

http://docs.oracle.com/javase/7/docs...ationInfo.html

[jvl711]

Originally Posted by Fuzzy

Are you seeing the delays only when running in the sagetv process, or also when your encoder is also in it's own JVM? There is a LOT more overhead going on in the sagetv JVM that may be blocking you out occasionally.

This is actually running outside of Sage. It has been driving me insane trying to figure it out.

[jvl711]

Originally Posted by EnterNoEscape

Perhaps you can subscribe to the garbage collection event and have it write out a log entry so you can attempt to correlate it. You brought something up that I had not considered might interfere with processing. I'll be keeping an eye out for that in my own implementation.

http://docs.oracle.com/javase/7/docs...ationInfo.html

I have reduced the Full GCs by increasing this parameter -XX:NewSize=50M. Copying the streams creates a ton of objects that are short lived. With default VM settings they were going to the Old Generation and causing full GCs to happen.

I have traced almost everything I could think of and saw no correlation to the pauses. Now I think there is a possibility that the pauses might have been coming from reading from STDIN to accept commands from the user. I am testing with that code removed for a while to see if that is where the issue is. I will report back.

Josh

[nyplayer]

Will the PrimeNetEncoder be able to handle MPEG-4/h.264 video ... Xfinity/Comcast in my area is changing over to this format beginning 10/25.

Nick stated that the HDHomerun primes already have the firmware upgrade to handle this conversion.

[uberpixel]

Originally Posted by nyplayer

Will the PrimeNetEncoder be able to handle MPEG-4/h.264 video ... Xfinity/Comcast in my area is changing over to this format beginning 10/25.

Nick stated that the HDHomerun primes already have the firmware upgrade to handle this conversion.

+1

Same here. I think we're neighbors.

-uberpixel

[nyplayer]

Originally Posted by uberpixel

+1

Same here. I think we're neighbors.

-uberpixel

I wonder what the new boxes look like.???

[uberpixel]

Originally Posted by nyplayer

I wonder what the new boxes look like.???

Can't say I care much about the appearance as long as they don't follow the annoying new trend of fashionable enclosures that make it impossible to stack with other components, or mount to a wall.

Most important feature for me would be Component out (although I stopped using analog capture a while ago). I only have the STB as a backup in case my server goes down or they decide to copy protect everything without notice.

-uberpixel

[weeber]

I thought Comcast was going to swap my market (Atlanta) to MPEG4 in August, but they still haven't. My old box (Motorola DCT) was not compatible with MPEG4.

Before I got the Prime and started using Primenetencoder, I tried the Comcast non-DVR box and it was a Pace RNG 110. It does have component. It also has a firewire port, but comcast disable video output on it and I could not use it to record like on my old box.

Just FYI, I believe all Comcast boxes are now Residential Network Gateways (RNGs)
http://www.dslreports.com/faq/17171

When Comcast finally switches over to MPEG4, I'll update and let everyone if Primenetencoder works for me. I'm keeping my fingers crossed it will work without a hitch.

Originally Posted by nyplayer

I wonder what the new boxes look like.???

[LazyGun]

Hello,

I have been trying to get this to work and not succeeding.

I have a new HDHomeRunPrime. I have tested that it works outside of Sage, it plays TV just fine on my PS3.
Within Sage I see all three tuners, and I can configure them. Sage says they are functioning.
However, I get No Signal when I watch TV.
The file in the recording folder is not growing in size, just sits at 0 bytes.
I have followed the instructions (and double checked them a few times too), and am pretty certain it is all configured ok.
The Tuner log just says:

10/02/2015 13:01:04 - Start commmand received: START PrimeNetEncoder 1321CEC6-0 Digital TV Tuner|343008772|804|2887610529074|\\zeus\sage\AccessHollywood-S20E22-33162424-1.ts|Great
10/02/2015 13:01:04 - -------------------------------------------------------------------------------
10/02/2015 13:01:04 - Switching Channel for Tuner: 1321CEC6 0
10/02/2015 13:01:04 - -------------------------------------------------------------------------------
10/02/2015 13:01:04 - Channel: 804
10/02/2015 13:01:04 - File: AccessHollywood-S20E22-33162424-1.ts
10/02/2015 13:01:04 - UploadID: 343008772
10/02/2015 13:01:04 - Quality: Great
10/02/2015 13:01:04 - Local IP: 10.0.0.5
10/02/2015 13:01:04 - Listening Port: 5000
10/02/2015 13:01:04 - Stream listening Port: 7000
10/02/2015 13:01:04 - Checking to see if the tuner is locked.
10/02/2015 13:01:04 - Switch channel: 804
10/02/2015 13:01:04 - 	Using Lockkey: 496916022
10/02/2015 13:01:04 - 	Command output: null
10/02/2015 13:01:04 - Send stream to UDP port: 7000
10/02/2015 13:01:04 - 	Using Lockkey: 496916022
10/02/2015 13:01:04 - 	Command output: null
10/02/2015 13:01:04 - Passing stream to SageTV unaltered (ffmpeg stream copy)
10/02/2015 13:01:04 - Starting TunerOutput thread for stdin to ffmpeg then stdout to SageTV MediaServer
10/02/2015 13:01:04 - Tuner output thread constructed for UploadID: 343008772
10/02/2015 13:01:04 - Tuner output thread HDHomeRun(UDP) -> PrimeNetEncoder(SDIN) -> ffmpeg(STDOUT) -> SageTV Media Server(TCP)
10/02/2015 13:01:04 - Sleeping to allow ffmpeg to fully launch: 1000
10/02/2015 13:01:04 - Sending write open command
10/02/2015 13:01:04 - TunerBridge thread started udpPort: 7000
10/02/2015 13:01:04 - Tuner bridge has transfered: 0
10/02/2015 13:01:04 - Write open sent successfully
10/02/2015 13:01:04 - Tuner bridge has transfered: 189504
10/02/2015 13:01:05 - Tuner bridge has transfered: 480340
10/02/2015 13:01:05 - Tuner bridge has transfered: 482972
10/02/2015 13:01:05 - Tuner bridge has transfered: 486920
10/02/2015 13:01:06 - Tuner bridge has transfered: 490868
10/02/2015 13:01:06 - Tuner bridge has transfered: 497448
10/02/2015 13:01:07 - Tuner bridge has transfered: 501396
10/02/2015 13:01:07 - Tuner bridge has transfered: 504028
10/02/2015 13:01:07 - Tuner bridge has transfered: 506660
10/02/2015 13:01:07 - Tuner bridge has transfered: 509292
10/02/2015 13:01:08 - Tuner bridge has transfered: 511924
10/02/2015 13:01:08 - Tuner bridge has transfered: 514556
10/02/2015 13:01:08 - Tuner bridge has transfered: 517188
10/02/2015 13:01:08 - Tuner bridge has transfered: 519820
10/02/2015 13:01:09 - Tuner bridge has transfered: 522452
10/02/2015 13:01:09 - Tuner bridge has transfered: 525084
10/02/2015 13:01:09 - Tuner bridge has transfered: 527716
10/02/2015 13:01:09 - Tuner bridge has transfered: 530348
10/02/2015 13:01:10 - Tuner bridge has transfered: 532980
10/02/2015 13:01:10 - Tuner bridge has transfered: 535612
10/02/2015 13:01:10 - No data transfered in 6000ms.  Reseting tuner channel and stream.

And this repeats
(Ignore the choice of program )

I've attached the .properties file as well. I did changed the ffmpeg delaysetting to 1000 (from 500), and I saw that the probesize was already set to 5000000 (I saw in a previous post that someone had upped this to this value to resolve this sort of issue).

Any hints or tips would be appreciated, thanks.

Also, thank you for putting this tool together.

Cheers

[nyplayer]

Have you tried it with the Firewall off ... that is one of the things I recommend doing first so we know if it is or isn't a firewall issue. Or were you just being a Lazy Gun lol?

FMMPEG.exe and JAVA.exe need access through the firewall also not just the ports.

[LazyGun]

Originally Posted by nyplayer

Have you tried it with the Firewall off ... that is one of the things I recommend doing first so we know if it is or isn't a firewall issue. Or were you just being a Lazy Gun lol?

FMMPEG and JAVA.exe need access through the firewall also not just the ports.

Sorry, forgot to mention that. The firewall is off already.
Thanks

(I am lazy though )

[nyplayer]

Originally Posted by LazyGun

Sorry, forgot to mention that. The firewall is off already.
Thanks

(I am lazy though )

Post all your tuner logs in the PrimeNetEncoder folder... and Sage properties

[LazyGun]

Originally Posted by nyplayer

Post your tuner logs in the PrimeNetEncoder folder.

You got it!

I have mainly tried tuner 0, but I also gave tuner 1 a shot as well. Same thing.
Tuner 0 you can see I was messing around a bit last night as well.

Thanks

[Edit] Added sage properties [/Edit]

[nyplayer]

Originally Posted by LazyGun

You got it!

I have mainly tried tuner 0, but I also gave tuner 1 a shot as well. Same thing.
Tuner 0 you can see I was messing around a bit last night as well.

Thanks

[Edit] Added sage properties [/Edit]

You sure that channel is not Protected SageTV cannot record protected channels.

PS Your ports in the Sage Properties appear to be reversed.... Did you let SageTV auto discover them ?

this one is definitely wrong.
mmc/encoders/-552182259/encoder_host=10.0.0.5\:7002
mmc/encoders/-552182259/encoder_merit=0
mmc/encoders/-552182259/encoding_host=10.0.0.5\:7002
mmc/encoders/-552182259/fast_network_encoder_switch=false
mmc/encoders/-552182259/forced_video_storage_path_prefix=
mmc/encoders/-552182259/last_cross_index=0
mmc/encoders/-552182259/last_cross_type=100
mmc/encoders/-552182259/live_audio_input=
mmc/encoders/-552182259/multicast_host=
mmc/encoders/-552182259/never_stop_encoding=false
mmc/encoders/-552182259/video_capture_device_name=PrimeNetEncoder 1321CEC6-2