OpenDCT: An open source digital cable tuner.

[waylo]

Originally Posted by cat6man

hi there........and thanks for the great work on openDCT.

i moved my old v7 server on windows xp to v9 on ubuntu and it is working very nicely. when i update openDCT, do i just stop the sage and openDCT services, run the installer, and re-start the services? (i.e. i don't have to re-do the console-only procedure or re-configure tuners, correct?)

Provided Sage is not recording anything, you can just stop the OpenDCT service, run the installer, and restart the service. You should probably do a test recording right afterwards. I've had it either run just fine OR claim that no encoder is available for that station. If the latter is the case, a simple 'configure video source' and channel preview is enough to get SageTV recognizing it again.

You do not need to redo the console procedure.

[EnterNoEscape]

Originally Posted by troll5501

I'll PM you with a link to several raw clips also, just in case my scenario is different than the others. Last night the tolerance incremented as high as 14400000 before it stopped making adjustments. And I'll give you last night's logfile in case it has any helpful information.

Thanks. Fortunately the changes I made also worked on those. The reason why the adjustments get so high is because when the streams are forced to sync, the distance between the last frame is still an issue, so that distance is added again. That's how ~-400000 becomes ~-800000 and so on. The worst apparently normal distance I ever saw was in some TV show I recorded in the morning. Only one stream had a difference of a little over -3000000. Everything recovered without too much disruption, but I think I'll use that as the benchmark and set the tolerance at 3500000 clock cycles (at 90000Hz, that's almost 39 seconds).

Both of your files and the one provided by CptKirk have the same spacing of the video frames. Which is packed a little strangely. It's worth noting that these Music Choice files mostly only playback well for me on the extenders. I think VLC was the only program I have installed that could play them back without hanging occasionally. For giggles I tried transcoding the streams. All 15 minutes were transcoded in 56 seconds. The resulting stream played back without issue on all of my players.

[EnterNoEscape]

• Changed starting dts tolerance from 450000 cycles to 3500000 to support the large gaps in Music Choice and other quirky channels.

[troll5501]

Originally Posted by EnterNoEscape

Thanks. Fortunately the changes I made also worked on those. The reason why the adjustments get so high is because when the streams are forced to sync, the distance between the last frame is still an issue, so that distance is added again. That's how ~-400000 becomes ~-800000 and so on. The worst apparently normal distance I ever saw was in some TV show I recorded in the morning. Only one stream had a difference of a little over -3000000. Everything recovered without too much disruption, but I think I'll use that as the benchmark and set the tolerance at 3500000 clock cycles (at 90000Hz, that's almost 39 seconds).

Both of your files and the one provided by CptKirk have the same spacing of the video frames. Which is packed a little strangely. It's worth noting that these Music Choice files mostly only playback well for me on the extenders. I think VLC was the only program I have installed that could play them back without hanging occasionally. For giggles I tried transcoding the streams. All 15 minutes were transcoded in 56 seconds. The resulting stream played back without issue on all of my players.

Great! Glad to hear this was an easy fix.

[cat6man]

Originally Posted by waylo

Provided Sage is not recording anything, you can just stop the OpenDCT service, run the installer, and restart the service. You should probably do a test recording right afterwards. I've had it either run just fine OR claim that no encoder is available for that station. If the latter is the case, a simple 'configure video source' and channel preview is enough to get SageTV recognizing it again.

You do not need to redo the console procedure.

thanks.
works brilliantly.

[nyplayer]

Music channels still have a problem plays for about 30 seconds then stops.

logs attached

[EnterNoEscape]

Originally Posted by nyplayer

Music channels still have a problem plays for about 30 seconds then stops.

logs attached

So few players seem to be able to work with the video in the format it's provided to the point that it breaks playback. Even the extenders at times can't deal with it, but they usually do for me at least. I really think these streams are at the edge of what's actually playable. There are zero errors/corrections in your log file. The only way that it will playback ok on those devices that can't take it as it comes in is transcoding which based on my testing of those channels is very light on the computer. Maybe I'll write a transcoding profile that should only match Music Choice content, but it will come with the 0.5 release because I don't know if I'll need to add some new capabilities to be it doesn't get false positives.

Update: I wrote a profile that should match only content that's formatted like Music Choice and I noticed something interesting about the SageTV Windows client. While transcoding and listening at the live-most edge of the recording, the sound cuts out occasionally. If I seek back a few seconds, eventually I end up at live again. It's the players job to figure out the jitter (difference between the player clock and source clock), but it's clearly not syncing very well with this content which is causing it to get too far ahead resulting in the audio cutting out. This doesn't appear to happen with the placeshifter or extenders. I just thought this might give everyone a little insight into what might be going on.

[cat6man]

Originally Posted by cat6man

thanks.
works brilliantly.

well, installed and records fine.
seems like liveTV crashes within a minute or two.
i'll get a log.

[EnterNoEscape]

Originally Posted by cat6man

well, installed and records fine.
seems like liveTV crashes within a minute or two.
i'll get a log.

It's strange that there would be any distinction. The only real difference between live and recording is when it's live, SageTV asks for the current file size more frequently.

[EnterNoEscape]

• CCExtractor support for FFmpeg transcoder has been added.
• Added opendct.consumer.DynamicConsumerImpl as the new default consumer. To change over to the new consumer from a previous installation, change the values of sagetv.device.<unique_id>.consumer to opendct.consumer.DynamicConsumerImpl.
• InfiniTV devices now have their own capture device and tuning via UPnP is considered deprecated. Discovery is still via UPnP.
• UPnP detection is now on demand by default similar to how the HDHomeRun detection currently works. This cuts down on traffic.
• New HDHomeRun lineups are now assigned the device id instead of IP address so if the IP address changes, the lineup knows where to look even if the IP address changes.
• All devices will now update their IP address if it changes.
• The internal circular buffer is now based completely off heap. The raw stream data now never enters the JVM. The circular buffers are also recycled so they don't need to be re-allocated on start and stop.
• Devices that use HTTP for streaming now use a direct byte buffer for communications by default. To move to the new communications method, change sagetv.device.parent.<parent_id>.http.producer to opendct.producer.NIOHTTPProducerImpl.
• Removed EIA information from channels since it is never used and just wasting RAM.
• Changed Linux priority from -19 to -5 since -19 puts the process higher than some network processes which might be a bad idea.
• Too many to list optimizations for the remuxer to greatly reduce object creation resulting in a measurable increase in overall throughput.
• Trying out fastutils for some of the bigger hash maps where possible.
• Added Music Choice transcoding profile (ultrafast_mc). This matches Music Choice channels based on their relatively unique characteristics. It is possible to have an SD channel come in as a false positive, but it should be rare.

CCExtractor Options:

• consumer.ffmpeg.ccextractor_all_streams=true

  This enables extracting of both CC1 and CC2. If this is off, it will output only CC1. You will likely want to leave this set to true.

• consumer.ffmpeg.ccextractor_custom_options=

  This can be any combination of valid CCExtractor options with the exception of -12, -1, -2, -in=, -out= and -o. Some people might want to change how roll-up captions are written for example. These options are sanitized to prevent execution of other programs, so you will not be able to do that from this option.

• consumer.ffmpeg.ccextractor_enabled=false

  This enables CCExtractor when set to true.

Dynamic Consumer Options:

• Channel Range Notation

  Channel ranges are defined by a number then a hyphen followed by another number (8-10). Multiple ranges are separated by a comma (13-15,8-10). ATSC channels are defined by a number (frequency channel) then a hyphen followed by a the channel number then a hyphen followed by the sub-channel number (50-8-1). The use of two hyphens automatically makes it literally a channel instead of a range.

  Ex. 8-10,50-8-1,13-16 will match the following channels: 8, 9, 10, 50-8-1, 13, 14, 15, 16

• consumer.dynamic.channels.ffmpeg=

  These are the channels that will always be assigned to the transcoding FFmpeg consumer.

• consumer.dynamic.channels.raw=

  These are the channels that will always be assigned to the raw consumer.

• consumer.dynamic.default=opendct.consumer.FFmpegTransSageTVConsumerImpl

  This is the consumer always used if the channel currently tuned doesn't match the channels under consumer.dynamic.channels.ffmpeg or consumer.dynamic.channels.raw.

[waylo]

Lots to explore with this new release!

For the new options you list, we need to manually add those lines into the properties file for an existing install, correct? I did not find them upon searching the text after install.

Is there something specific needed to activate the ultrafast_mc profile?

EDIT:
Music channels still would not properly play with the default trans consumer. However, with the dynamic channel set to raw for them, they work great.
Have not tried CCextractor yet.

[AlphaCrew]

What are the advantages of OpenDCT over SageDCT? My SageDCT is working well, with very little cpu consumption.

[EnterNoEscape]

Originally Posted by waylo

Lots to explore with this new release!

For the new options you list, we need to manually add those lines into the properties file for an existing install, correct? I did not find them upon searching the text after install.

Is there something specific needed to activate the ultrafast_mc profile?

EDIT:
Music channels still would not properly play with the default trans consumer. However, with the dynamic channel set to raw for them, they work great.
Have not tried CCextractor yet.

A transcoding profile is by default not configured which translates into always remux. It probably would have been helpful to comment on how to use that profile since it's been a while since it was mentioned in the release notes and the first post didn't even mention anything about transcoding. I fixed that.

Set the software transcoding profile by changing the values of sagetv.device.<unique_id>.transcode_profile to one of the available profiles are stored under C:\ProgramData\OpenDCT\config\transcode on Windows and /etc/opendct/conf/transcode on Linux. See profile_example.properties for help on how you can create your own. Do not include the .properties extension when setting the profile.


Raw is probably always going to be the fastest option for Music Choice. The problem is that FFmpeg needs to know a lot more about the video and audio than is actually needed to correctly remux them and collecting that information takes longer with fewer packets.

[cat6man]

Originally Posted by EnterNoEscape

It's strange that there would be any distinction. The only real difference between live and recording is when it's live, SageTV asks for the current file size more frequently.

i'll take another look and get log files after i update the consumer.
a difference between the two cases could be that it is crashing the hd300, which doesn't come into play with recording and does with liveTV.

[nyplayer]

Something I noticed that when using OpenDCT you can delete a show that is recording from within file explorer and it just deletes it does not say in use. It seems that OPenDCT does not hold the file so it cannot be modified? is this by design? I can even rename it then it just starts a new recording. It should only be available while recording as read only.


EDIT I am in the process of adding some jobs that move recordings first it checks for use .... but it always says not in use so it moves the partial file then SageTV creates a new one.

[brandypuff]

Writes are probably buffered and appended in chunks.

[nyplayer]

Originally Posted by brandypuff

Writes are probably buffered and appended in chunks.

Could this cause a problem with Comskip LiveTV detection if it thinks it has reached the EOF?. I changed my move jobs to check for 5 minutes old files at the minimum.

[nyplayer]

Now on another note the Subtitles work great ... Thanks for implementing this.... even works while recording that I didn't expect is a pleasant surprise.

Edit no problem with Comskip LiveTV just had to change my move jobs by age not by use.

[EnterNoEscape]

Originally Posted by nyplayer

Now on another note the Subtitles work great ... Thanks for implementing this.... even works while recording that I didn't expect is a pleasant surprise.

Edit no problem with Comskip LiveTV just had to change my move jobs by age not by use.

I'm glad to hear the subtitles are working well on currently recording content. The only thing you'll find annoying about live recordings is that the subtitles will suddenly cut off at wherever they were when you started watching. You would then need to stop the recording and start it again to see the most recently written subtitles.

Live Comskip works by seeing if the file has grown. There's a property that basically defines how many times in I believe are 1 seconds intervals to check if the file has grown before it gives up and assumes the recording must be done. If the file does grow, the counter resets.

OpenDCT handles a file missing situation by assuming it needs to re-create the file. I knew that in Linux you could delete the file right from underneath it, but I never thought to try it with Windows. I just assumed that since it has a handle for the file open, that it would also prevent the file from being deleted. I know that renaming is often possible even when a file is open in either Windows or Linux. The most interesting thing is in neither Windows or Linux does the method used for writing throw any exceptions if the file suddenly goes missing which is why OpenDCT periodically checks for the presence of the file.

I could try to ensure the files are read-only while recording if it might be of benefit to anyone. However in Linux-land, you can't really ensure that someone logged in as root won't be able to delete the file anyway.

[nyplayer]

Originally Posted by EnterNoEscape

I'm glad to hear the subtitles are working well on currently recording content. The only thing you'll find annoying about live recordings is that the subtitles will suddenly cut off at wherever they were when you started watching. You would then need to stop the recording and start it again to see the most recently written subtitles.

Live Comskip works by seeing if the file has grown. There's a property that basically defines how many times in I believe are 1 seconds intervals to check if the file has grown before it gives up and assumes the recording must be done. If the file does grow, the counter resets.

OpenDCT handles a file missing situation by assuming it needs to re-create the file. I knew that in Linux you could delete the file right from underneath it, but I never thought to try it with Windows. I just assumed that since it has a handle for the file open, that it would also prevent the file from being deleted. I know that renaming is often possible even when a file is open in either Windows or Linux. The most interesting thing is in neither Windows or Linux does the method used for writing throw any exceptions if the file suddenly goes missing which is why OpenDCT periodically checks for the presence of the file.

I could try to ensure the files are read-only while recording if it might be of benefit to anyone. However in Linux-land, you can't really ensure that someone logged in as root won't be able to delete the file anyway.

No big deal on the file in use I did some further checking and that is exactly how the HDHomerun DVR does it. The files can be deleted or renamed.