PrimeNetEncoder - HDHomeRun Prime Network Encoder

[uberpixel]

Originally Posted by mayamaniac

Looks like I spoke to soon. Switching mediaserver.transfer=false reduces the write errors glitches, but not completely. It's about 80% less, but that's with one recording going. When 2 or 3 going at the same time, the glitches are 10 times worse. This problem is only with the HDHR Prime recordings, the regular HDHR OTA tuner records without issues.

I'm seeing the same thing on my WHS 2011 server. I noticed a significant uptick in artifacts and faults after upgrading to 1.2.5. I changed the mediaserver.transfer to false and noticed a big improvement but continue to see artifacts during heavy recording scenarios. I haven't had any actual faults since switching to false which represents a significant improvement. Still seeing artifacts though.

Let me know if there is any information I can share to help with troubleshooting. All of my (other) properties are set to their defaults.

-uberpixel

[Fuzzy]

The mediaserver is only the very last step of the chain, and is not really the most finicky. The toughest part of the chain is the piping of the data from PNE to ffmpeg, then piping it back to PNE - which is likely where the info is getting lost.

[stuckless]

Originally Posted by Fuzzy

The mediaserver is only the very last step of the chain, and is not really the most finicky. The toughest part of the chain is the piping of the data from PNE to ffmpeg, then piping it back to PNE - which is likely where the info is getting lost.

I think I mentioned before that on the Android miniclient, I ran into performance issues piping data between processes in Java. I was using PipedInputStream but the performance was bad. I don't know what you guys are using, but if you are using PipedInputSTream it might be causing some issues. Is the code for this on github? Or is this a closed source application?

[mayamaniac]

How is SageDCT doing it?

[Fuzzy]

Originally Posted by mayamaniac

How is SageDCT doing it?

Depends on what method is being used. Dump mode takes the stream direct from the HDHR, and writes it to disk (no piping between processes). ffmpeg mode, I believe, tells ffmpeg to take the stream directly from the HDHR and write it to the disk (no piping either).

[jgsouthard]

Originally Posted by jvl711

The file writing method is a good theory. In version 1.0.1 ffmpeg was doing the writing. In 1.2.5 SageTV medianserver is doing thwe file writing. I am not sure how well tested the option is, but you could try setting mediaserver.transfer to false. In that case either PrimeNetEncoder or ffmpeg would handle writing.

My PNE implementation is batch file via Task Scheduler on Windows startup, priority at 4. One HDHomeRun Prime (3 tuners). SageTV running headless on one Windows 10 PC, feeding three HD300 media extenders.

I've been getting occasionally (but frequent enough to be irritating) "halt detected in recording" errors with PNE 1.0.1, particularly at the beginning of recordings. Most of the time I just lost the first minute or two of the recording, after which it worked OK other than occasional artifacts (also irritatingly frequent).

So when 1.2.5 came out, I switched to it in the hope that I could get rid of these errors. I modified the Java batch file with the recommended parameters, removed all of the mmc encoder lines from Sage.properties, and started from scratch with new video source discovery in SageTV.

While installation went smoothly, my halt errors increased significantly to the point that SageTV was unusable (losing desired recordings daily). Rather than just at the beginning of recordings, I was getting numerous errors and losing the first 10-15 minutes of recordings, or in some cases the entire recording.

Last night, based on these recent comments in the last couple of days, I changed mediaserver.transfer=true to mediaserver.transfer=false in primenetencoder.properties. So far, no halts on about 6-8 recordings.

I'll give it another day or two before declaring success, since in the past it has seemed to work well for 24 hours or so and then get worse over time until a restart. But so far I'm encouraged.

Bottom line: PNE 1.2.5 as configured did not work for me at all, with a far higher rate of halt errors than I was getting with 1.0.1. So if changing mediaserver.transfer helps, I think that change needs to be set by default. And there may still be something about 1.2.5 that is actually a step backward rather than forward.

[jvl711]

Originally Posted by jgsouthard

My PNE implementation is batch file via Task Scheduler on Windows startup, priority at 4. One HDHomeRun Prime (3 tuners). SageTV running headless on one Windows 10 PC, feeding three HD300 media extenders.

I've been getting occasionally (but frequent enough to be irritating) "halt detected in recording" errors with PNE 1.0.1, particularly at the beginning of recordings. Most of the time I just lost the first minute or two of the recording, after which it worked OK other than occasional artifacts (also irritatingly frequent).

So when 1.2.5 came out, I switched to it in the hope that I could get rid of these errors. I modified the Java batch file with the recommended parameters, removed all of the mmc encoder lines from Sage.properties, and started from scratch with new video source discovery in SageTV.

While installation went smoothly, my halt errors increased significantly to the point that SageTV was unusable (losing desired recordings daily). Rather than just at the beginning of recordings, I was getting numerous errors and losing the first 10-15 minutes of recordings, or in some cases the entire recording.

Last night, based on these recent comments in the last couple of days, I changed mediaserver.transfer=true to mediaserver.transfer=false in primenetencoder.properties. So far, no halts on about 6-8 recordings.

I'll give it another day or two before declaring success, since in the past it has seemed to work well for 24 hours or so and then get worse over time until a restart. But so far I'm encouraged.

Bottom line: PNE 1.2.5 as configured did not work for me at all, with a far higher rate of halt errors than I was getting with 1.0.1. So if changing mediaserver.transfer helps, I think that change needs to be set by default. And there may still be something about 1.2.5 that is actually a step backward rather than forward.

Can you please post your settings file, and log files from around the time a halt occurs. I would like to see if I see anything odd.

FYI... Your symptoms sound like you may have network performance issues. You may want to try a packet loss test with the HDHomeRun config app. Version 1.0.1 should have had 0 glitches/hiccups. If you were seeing them then it could be network related.

[jgsouthard]

Originally Posted by jvl711

Can you please post your settings file, and log files from around the time a halt occurs. I would like to see if I see anything odd.

FYI... Your symptoms sound like you may have network performance issues. You may want to try a packet loss test with the HDHomeRun config app. Version 1.0.1 should have had 0 glitches/hiccups. If you were seeing them then it could be network related.

jvl711,

Thanks for your development work on PrimeNetEncoder. Even though I may be running into issues, my wife and I are still using SageTV daily thanks to your help.

I'm not familiar with how to run a packet loss test with any of the utilities available with HDHomeRun -- Config, Setup, View. Could you point me in the right direction? I do large backups nightly over our LAN without packet loss (according to the stats in my Linux-based file server), so I think the network is solid; but I leave open the possibility that there could be an issue with the NIC in the SageTV box.

Here's my PrimeNetEncoder batch file, run at Windows startup from Task Scheduler with priority 4 (Normal):

CD /d "%~dp0"
:loop
java -XX:-UseBiasedLocking -XX:NewSize=50M -Xms128m -Xmx256m -jar PrimeNetEncoder.jar
goto loop

Here's my PrimeNetEncoder.properties file:

ffmpeg.delay=1500  <-- Default was 500, also tried 1,000, didn't seem to matter
ffmpeg.probesize=5000000
ffmpeg.analyzeduration=5000000
ffmpeg.path=ffmpeg.exe
HDHomeRunConfig.path=C:\Program Files\Silicondust\HDHomeRun\hdhomerun_config.exe
tuners.count=3
discovery.port=8271
discovery.enabled=true
mediaserver.transfer=false  <-- LOTS of Halt error with 'true,' none so far with 'false'
ffmpeg.outputbuffersize=32768
ffmpeg.inputbuffersize=32768
mediaserver.outputbuffersize=32768
mediaserver.transfer=false
tuner0.name=PrimeNetEncoder 1319784E-0
tuner0.id=1319784E
tuner0.number=0
tuner0.port=7000
tuner0.transcoder.port=5000
tuner0.enabled=true
tuner0.transcode.enabled=false
tuner0.transcode.deinterlace=true
tuner0.transcode.bitrate=4000
tuner0.transcode.scaling=
tuner0.transcode.preset=veryfast
tuner0.transcode.codec=libx264
tuner1.name=PrimeNetEncoder 1319784E-1
tuner1.id=1319784E
tuner1.number=1
tuner1.port=7001
tuner1.transcoder.port=5001
tuner1.enabled=true
tuner1.transcode.enabled=false
tuner1.transcode.deinterlace=true
tuner1.transcode.bitrate=4000
tuner1.transcode.scaling=
tuner1.transcode.preset=veryfast
tuner1.transcode.codec=libx264
tuner2.name=PrimeNetEncoder 1319784E-2
tuner2.id=1319784E
tuner2.number=2
tuner2.port=7002
tuner2.transcoder.port=5002
tuner2.enabled=true
tuner2.transcode.enabled=false
tuner2.transcode.deinterlace=true
tuner2.transcode.bitrate=4000
tuner2.transcode.scaling=
tuner2.transcode.preset=veryfast
tuner2.transcode.codec=libx264

Since I changed my mediaserver.transfer setting to false about 24 hours ago I have had no further halt errors, and I've restarted SageTV enough times that all of the old logs with errors have already rolled off. I changed the setting back briefly tonight, but I've run out of time before desired recordings start and haven't been able to generate another error. I'll keep watching and try to capture something in the log.

[jgsouthard]

OK, I found instructions for running a packet loss test on HDHomeRun Prime at SiliconDust. I ran it for about 10-15 minutes (all the time I had before the next recording) with zero errors in over 500K packets transferred.

[jvl711]

Originally Posted by jgsouthard

OK, I found instructions for running a packet loss test on HDHomeRun Prime at SiliconDust. I ran it for about 10-15 minutes (all the time I had before the next recording) with zero errors in over 500K packets transferred.

Can you try having watching live TV, and then doing packet loss test on a free tuner.

The reason I am thinking there may be a network issue is that when you use the media server transfer option you are adding extra network overhead. You are technically doubling the amount of network traffic.

[nyplayer]

Originally Posted by jvl711

Can you try having watching live TV, and then doing packet loss test on a free tuner.

The reason I am thinking there may be a network issue is that when you use the media server transfer option you are adding extra network overhead. You are technically doubling the amount of network traffic.

If you are doubling the Network Traffic what benefit does enabling media server transfer have? I would think that doubling the Network traffic can only lead to problems.

[Fuzzy]

It's not doubling real network throughput, as it should be routing the extra bandwidth through the localhost.

[nyplayer]

Originally Posted by Fuzzy

It's not doubling real network throughput, as it should be routing the extra bandwidth through the localhost.

But what are the benefits if enabled???

[jvl711]

Originally Posted by nyplayer

If you are doubling the Network Traffic what benefit does enabling media server transfer have? I would think that doubling the Network traffic can only lead to problems.

Good question:

1. Simplification of setup for the end user. If PrimeNetEncoder is on another machine than the server, the user does not have to think about how the machine is going to access the filesystem that Sage is using for recordings.

2. Most of the changes in this version were intended to "hopefully" move us to having Sage do the remuxing instead of ffmpeg. I am attempting to add remuxing support directly into Sage Media Server. I wanted to have media server transfer working well before that potential change. Technically all the code required for doing that is already in PrimeNetEncoder now. I am using this version to test the new functionality in Sage.

3. I think channel changes may be a hair quicker. This is subjective, and I have no evidence to back this.

It may make sense to make false the default option if we are getting a lot of people having issues with it. I am not sure how many people are using PrimeNetEncoder to know if it is a large issue or not. So far I think only two users have set it to false. I would like to find out what the issue is with these users, because in the future I am hoping the Sage remuxer will be the better option, and if they are not able to use media server transfer they will not be able to use that feature.

Fuzzy is correct, it probably is not doubling network use, but there is additional strain on the network stack, because it is dealing with 2 times the number of packets. There just is not 2 times the packets going over the wire.

On the positive side... It is very easy to turn off

[uberpixel]

Originally Posted by jvl711

So far I think only two users have set it to false. I would like to find out what the issue is with these users, because in the future I am hoping the Sage remuxer will be the better option, and if they are not able to use media server transfer they will not be able to use that feature.

I am one of those known users with ongoing issues. Let me know what data I can provide to help with your troubleshooting. I'm not an IT or coding professional, so you may have to dumb down your requests.

Just to recap, my system was relatively stable with the previous version (occasional halts and artifacts as well as completely losing the tuners from the lineup). After switching to 1.2.5, I noticed worse performance in all of those categories until I switched the mediaserver.transfer to false. With it set to false I still see artifacts and halts under heavy recording scenarios (prime time) and also during middle of the night recording (usually single tuner only) which may coincide with various scheduled backups (WHS client and server backups and Crashplan).

The late night issues do not appear to coincide with the scheduled start times of any of the backup operations because the halts typically occur at non-round numbered times (not at the top of the hour or half hour) and most of my scheduled events are set to start at rounded numbers (Crashplan may be more unpredictable). The halts may coincide with the backups, but the start does not seem to trigger an immediate reaction.

That said, one would expect that there would be increased demand on the server during these scenarios so it may be worth more investigation. My dual TV recording drives are dedicated for only SageTV use so I doubt that they are getting overwhelmed. My network is gigabit LAN but could be a potential bottleneck I suppose? I have dual onboard NIC's. Would there be any benefit to splitting the traffic somehow?

This morning my server could not find the Prime tuners. A system restart resolved the issue.

-uberpixel

[jvl711]

Originally Posted by uberpixel

This morning my server could not find the Prime tuners. A system restart resolved the issue.

-uberpixel

Can you give a little more detail around this. What applications could not see the primes? If you open the hdhomerun_config app were you able to see the tuners?

Some of this might be greek to you but can you review some of the tips in this link

[nyplayer]

Originally Posted by uberpixel

I am one of those known users with ongoing issues. Let me know what data I can provide to help with your troubleshooting. I'm not an IT or coding professional, so you may have to dumb down your requests.

Just to recap, my system was relatively stable with the previous version (occasional halts and artifacts as well as completely losing the tuners from the lineup). After switching to 1.2.5, I noticed worse performance in all of those categories until I switched the mediaserver.transfer to false. With it set to false I still see artifacts and halts under heavy recording scenarios (prime time) and also during middle of the night recording (usually single tuner only) which may coincide with various scheduled backups (WHS client and server backups and Crashplan).

The late night issues do not appear to coincide with the scheduled start times of any of the backup operations because the halts typically occur at non-round numbered times (not at the top of the hour or half hour) and most of my scheduled events are set to start at rounded numbers (Crashplan may be more unpredictable). The halts may coincide with the backups, but the start does not seem to trigger an immediate reaction.

That said, one would expect that there would be increased demand on the server during these scenarios so it may be worth more investigation. My dual TV recording drives are dedicated for only SageTV use so I doubt that they are getting overwhelmed. My network is gigabit LAN but could be a potential bottleneck I suppose? I have dual onboard NIC's. Would there be any benefit to splitting the traffic somehow?

This morning my server could not find the Prime tuners. A system restart resolved the issue.

-uberpixel

You might want to read this thread on how to Optimize your NIC cards.

http://forums.sagetv.com/forums/showthread.php?t=61875

This also might help.

https://social.technet.microsoft.com...inserverhyperv

[uberpixel]

Originally Posted by jvl711

Can you give a little more detail around this. What applications could not see the primes? If you open the hdhomerun_config app were you able to see the tuners?

Some of this might be greek to you but can you review some of the tips in this link

When this happens I usually find out because a bunch of tuner conflicts show up out of nowhere. If I go to add video source, the Prime doesn't show up on the list. Pretty sure they are still functioning fine but I'm unable to easily view the results from HDHomerun viewer since OpenGL isn't supported via Remote Desktop. I will try the config app next time this happens. A system (server) restart usually fixes this, I just reboot the Prime now out of habit. It may be unnecessary.

I will explore some of the network related tests this weekend and report back.

-uberpixel

[uberpixel]

Originally Posted by uberpixel

I will explore some of the network related tests this weekend and report back.

Interesting discovery. LANSpeedTest shows that I am getting 800Mbps (gigabit) Upload but only 80Mbps Download speeds...

That's between my SageTV server machine and a client. When I go between two clients they both have ~800Mbps for both upload/download. So my assumption was that there is something screwy with the NIC on my server.

Here's the wrinkle - I tried the other NIC on my mobo and found the exact same situation. So, it doesn't necessarily seem like the NIC settings are wrong (unless they are wrong on both NIC's which I guess could be the case) - could it be something within WHS 2011 settings? I swapped cables and took a gigabit switch out of the mix and connected directly to my router and saw no improvement...

Not sure if that would impact PrimeNetEncoder performance though. What are your thoughts?

Either way, I want to resolve this (at the risk of getting off topic) so I can improve my server performance. Any ideas where to look for a setting change to fix this issue.

-uberpixel

[Fuzzy]

3rd party firewall or virus scanner molesting incoming data to the server?