ffmpeg development

[Fuzzy]

So, has anyone here done any ffmpeg/mplayer development? I ask, because currently, there is an stv.c file in sage's implementation of libavfilter, which allows reading of files from the sage server via sage's MediaServer architecture. I'd like to extend this protocol support to include writing to the MediaServer as well - such that you could output from ffmpeg directly to a waiting sagetv server via a stv://UploadKey@server/path/to/file.mpg ffmpeg command. This would greatly help in the performance of network encoders like primenetencoder by skipping some of the piping steps, especially if said encoder sources have the ability to stream their output directly into a supported ffmpeg protocol. Then, the network encoder is essentially just handling some management tasks.

So has anyone messed with ffmpeg protocol modules? I'm not a c++ guy, so looking through stv.c is a bit difficult for me to grasp.

[stuckless]

So, no on the ffmpeg part, but, is a network encoder the same thing as a network tuner? I was surprised (a couple years back) when I wrote a network tuner, and it was file based in terms of the linke between sagetv server and the network turner.

[Fuzzy]

Originally Posted by stuckless

So, no on the ffmpeg part, but, is a network encoder the same thing as a network tuner? I was surprised (a couple years back) when I wrote a network tuner, and it was file based in terms of the linke between sagetv server and the network turner.

File based is the 'simplest' method, but it requires that the encoder have write access to the shares, which limits it's use for embedded type network encoders. However, Sage also allows sending the file via packets to it's MediaServer port (this was never documented in the Network Encoder spec - but we can see and use it now with the source opened up). That is how the newer versions of PrimeNetEncoder are doing it, but he is having to grab the data from the prime via UDP, then pipe the data to the stdin of ffmpeg, then take the stdout of ffmpeg and pipe that back into PrimeNetEncoder, then send it to sage's MediaServer port. Makes for a finicky setup, and requires some special controls on the JVM to ensure error free throughput.

[Fuzzy]

the sage modified version of ffmpeg and mplayer (SageTVTranscoder.exe and SageTVPlayer.exe) support sage's mediaserver port, for reading only. You can show this by passing an stv:// url to SageTVTranscoder as such:

SageTVTranscoder.exe -i stv://sagetv/L:\Recordings_00\Tammy-23867682-0.ts

[stuckless]

Originally Posted by Fuzzy

File based is the 'simplest' method, but it requires that the encoder have write access to the shares, which limits it's use for embedded type network encoders. However, Sage also allows sending the file via packets to it's MediaServer port (this was never documented in the Network Encoder spec - but we can see and use it now with the source opened up). That is how the newer versions of PrimeNetEncoder are doing it, but he is having to grab the data from the prime via UDP, then pipe the data to the stdin of ffmpeg, then take the stdout of ffmpeg and pipe that back into PrimeNetEncoder, then send it to sage's MediaServer port. Makes for a finicky setup, and requires some special controls on the JVM to ensure error free throughput.

Originally Posted by Fuzzy

the sage modified version of ffmpeg and mplayer (SageTVTranscoder.exe and SageTVPlayer.exe) support sage's mediaserver port, for reading only. You can show this by passing an stv:// url to SageTVTranscoder as such:

SageTVTranscoder.exe -i stv://sagetv/L:\Recordings_00\Tammy-23867682-0.ts

I'm actually a lot more familiar with sage's push: and pull (stv protocols than I was a month ago, since the Android client has to implement both of these protocols for playing video

On the Android side, because of how the player libraries setup (ie, they are url based), then I have to create a http server that the LibVLC can use to play a video, that wraps the sagetv socket based protocol. So, like PrimeNetEncoder, I'm taking it in one stream and pushing it to another stream that is used by LibVLC. When I first created this I used Java's PipedInputStream and PipedOutputStream and the video playback was very poor. ie, LibVLC was depleting the stream faster than I could buffer it causing pauses every few milliseconds during playback. I eventually replaced java's PipedInputStream/OutputStream with another CircularByteArray library and the playback is great.

So, I'm not sure what the issue are with PrimeNetEncoder, if they are using PipedInputStream then perhaps they are hitting the same types of performance issues that I ran into on the Android side. Ideally when you are piping stdin to something else, you are using multiple threads and have very little blocking (if any) happing during the read/writes... the exception being that if you are using a memory buffer, then you need to block for the buffer to deplete enough bytes before writing more bytes into it. I think for the SageTV mplayer, it uses a circular file buffer to achieve the same thing.

I'm wondering if perhaps the Network Encoder API needs to have a Stream mode whereby sagetv can pull the stream from the encoder. I haven't looked at the NetworkEncoder api since the code has been open sourced, so maybe that is there, but if not, then I think it would be good addition for streaming. In this case maybe the encoder replies with a url (http://networkencoder/file, or stv://networkencoder/file, or file:///somesharedarea/file) and then sagetv can open the stream on the network encoder, and then write the file locally.

[Fuzzy]

The PrimeNetEncoder issues aren't so much an issue of the transport method between the encoder and the server, it's an issue of getting the data into and out of ffmpeg (which is used to remux the stream to clean up transmission issues). There's no 'clean' way of having ffmpeg handle this, so it is using the worst interface ffmpeg offers, stdin and stdout. I believe one of these pipes could be removed as I think there are methods of having ffmpeg pull directly from the HDHR, but there is no way to remove the other until ffmpeg can write to the MediaServer port.

For most users the primenetencoder is working fine, if in it's own JVM, with garbage collection tweaks in place, but there are some users on VM's that have had issues even with those tweaks. STDIN and STDOUT pipes are just not equipped to handling video bitrates without dropping data.

[Fuzzy]

The MediaServer.java has a decent enough buffer it seems - I've never seen it drop anything. Makes sense to have the encoder push to the server, since the encoder is the master of the bitstream anyway. In the tuning request (START), sage tells the encoder the sage-referenced filename, and an UploadID (as well as the encoder and channel, of course). The network encoder can then choose to write the file directly, or use the MediaServer push method. The UploadID works as a rudimentary key to allow an otherwise anonymous network source to store a file on the sage server.

[Fuzzy]

For my potential usage, I am working on an embeded network encoder for the HD-PVR. It's on very light hardware, and as such, would benefit from the lightest implementation. The need to handle http requests and such would be far moer complicated than simply pushing data to network ports while tracking byte count. But, I don't want to give up the ability to remux on the fly, so an ffmpeg implementation compiled for my ARM hardware, that included the WRITE capability in the stv.c module would be more than adequate.

[Narflex]

I've done a lot with FFMPEG and Mplayer as you might have guessed. I would never want to use FFMPEG as part of a network encoder/tuner..it's handling of transport streams always seemed pretty bad to me. Somebody else on the forums was working on using the remuxing logic inside of SageTV instead and making a command on the MediaServer to support streaming to it with automatic remuxing of the content. The remuxing system in SageTV is pretty solid IMHO and has worked great for a long time now...so I think using that is way better than FFMPEG.

Not sure where that project ended up though that was being worked on...

[jvl711]

Originally Posted by Narflex

I've done a lot with FFMPEG and Mplayer as you might have guessed. I would never want to use FFMPEG as part of a network encoder/tuner..it's handling of transport streams always seemed pretty bad to me. Somebody else on the forums was working on using the remuxing logic inside of SageTV instead and making a command on the MediaServer to support streaming to it with automatic remuxing of the content. The remuxing system in SageTV is pretty solid IMHO and has worked great for a long time now...so I think using that is way better than FFMPEG.

Not sure where that project ended up though that was being worked on...

Same guy is working on Both PrimeNetEncoder and the remuxing code in the media server

I did write an implementation in Sage MediaServer to do the remuxing using built in Sage remuxer. It works, but it does seem to have some issues. I switched focus back to PrimeNetEncoder for a little while. I wanted to change the architecture to possibly accommodate the remuxer being in Sage. I have now accomplished that part. I am reading from the HDHomeRunPrime directly, so I can now send streams directly to Sage. I plan to look at the remuxing code again when I free up. Below are the two issues I saw when I got it working.

1. It did not seem to release the file. Even when I properly closed the remuxer.

2. It occasionally crashed. If I changed channels a handful of times it would throw a jni error and crash sage. I am not sure if this is me or the library.

If there are bugs in the remuxer code I will probably be lost in fixing it though. I am not very good with c/c++. I am more of a Java/C# guy.

[Narflex]

Originally Posted by jvl711

1. It did not seem to release the file. Even when I properly closed the remuxer.

2. It occasionally crashed. If I changed channels a handful of times it would throw a jni error and crash sage. I am not sure if this is me or the library.

If there are bugs in the remuxer code I will probably be lost in fixing it though. I am not very good with c/c++. I am more of a Java/C# guy.

1. Should be easy to figure out and fix.

2. We can probably get Qian to help somewhat with this, he wrote all that native code and knows it very well. A reproducible case would make it much easier (i.e. save your input data to a file and then run the main() inside that class to reproduce it from file based input...which hopefully would still cause the same problem)

[jvl711]

Originally Posted by Narflex

1. Should be easy to figure out and fix.

2. We can probably get Qian to help somewhat with this, he wrote all that native code and knows it very well. A reproducible case would make it much easier (i.e. save your input data to a file and then run the main() inside that class to reproduce it from file based input...which hopefully would still cause the same problem)

I will try and modify the code so that it saves off the stream unmodified. Hopefully this will create a file that is able to reproduce the crash I am seeing. I will try and take a look at the code again tonight if I have some time.

Josh

[jvl711]

Attached is the error file that is generated when Sage crashes remuxing the stream. I am not sure if there is any info that could be used to determine the cause.

Josh

[Narflex]

Originally Posted by jvl711

Attached is the error file that is generated when Sage crashes remuxing the stream. I am not sure if there is any info that could be used to determine the cause.

Josh

Interesting...that crash is NOT in the remuxer...it's in the format detector code instead. Right before it crashes, it should print in the logs the file path it's doing the format detection on (it's likely the stream being created by the remuxing). I'd think you could reproduce this crash if you have the same file size as what it is when this fails (or maybe even bigger works)....however, it does things a little bit different if the recording is in progress as well I think.

Do you know how to write code in Java and C++? If so, then it shouldn't be too hard for you to generate a reproducible case of this which then will make it way easier to fix.

[jvl711]

Originally Posted by Narflex

Interesting...that crash is NOT in the remuxer...it's in the format detector code instead. Right before it crashes, it should print in the logs the file path it's doing the format detection on (it's likely the stream being created by the remuxing). I'd think you could reproduce this crash if you have the same file size as what it is when this fails (or maybe even bigger works)....however, it does things a little bit different if the recording is in progress as well I think.

Do you know how to write code in Java and C++? If so, then it shouldn't be too hard for you to generate a reproducible case of this which then will make it way easier to fix.

If I knew how to write C++ code I think I might have possibly figured out where the issue was coming from

Do you think my idea of writing the raw stream to a temp file along side of remuxing I would be able to recreate the crash, and have a temp file that should hopefully be able to reproduce the scenario?

Do you think that the format detector is not finding the appropriate format and possibly overflowing??? It did seem like sometimes it was detecting the format quickly, and other times it was taking a very long time.

Josh

[jvl711]

Originally Posted by Narflex

Interesting...that crash is NOT in the remuxer...it's in the format detector code instead. Right before it crashes, it should print in the logs the file path it's doing the format detection on (it's likely the stream being created by the remuxing). I'd think you could reproduce this crash if you have the same file size as what it is when this fails (or maybe even bigger works)....however, it does things a little bit different if the recording is in progress as well I think.

Do you know how to write code in Java and C++? If so, then it shouldn't be too hard for you to generate a reproducible case of this which then will make it way easier to fix.

Alright. I have written the code to save off a temp file with the data pushed into the remuxer to init it. I just need to repro the issue now where Sage crashes.

For some odd reason I am having a hard time repo-ing the issue using recordings instead of live tv. Is there any possibility the error is happening from the playback side of Sage instead of the remuxer stack? Would a different path be taken in the code based on Live TV vs. Recording.

Regardless... I will keep trying and repo.

[Narflex]

Remember what I said...the problem isn't the remuxer is crashing from the log you posted...it was the format detector. So you don't need the input to the remuxer in order to reproduce this crash...you want whatever was being fed into the format detector; and the sagetv_0.txt file should have that on the last line before the crash occurred.

However, when you try to reproduce it will likely fail because there's a flag that's different for format detection when something is recording. So I'd say to try the following in order to replicate it exactly:

1. Get the filename from the log of what's crashing the format detector
2. Make a copy of that file
3. Call sage.media.format.FormatParser.getMediaAVInf0("FilePathFromAbove", Min(fileLength, 30MB), true, -1) (just write a simple main() to call it, I'm assuming you know Java)

That then *should* crash because that's the same call it would have been making before that caused the crash.

[jvl711]

OK. Below is the end of the logfile. I think I might see what the problem might be. It appears that the remuxer has not determined the format yet, but Sage attempts to determine the format of the file. When this happens the actual file size is still 0 because the remuxer has not started remuxing yet. I am assuming there is a threshold on the GET_FILE_SIZE that tells sage there is enough data to determine the file type.

Is the problem coming from the fact that I am sending a file size back to Sage that is wrong from the network encoder. The network encoder is not currently aware of how far along the remuxer is. I am sending back bytes transferred for file size at the moment. Does this sound like it could be the issue? I guess I can send the fileSize command back to the media server to determine the size of the file. This could add some additional overhead to the transfer. Is there anyway to tell Sage to check the file size itself?

Let me know if you think I am off base, and I will double check to make sure the format parser can determine the video format of the file

Tue 11/3 19:09:47.774 [MediaServerConnection@217ee] MMC sent request to 192.168.0.65:7008 of GET_FILE_SIZE D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:47.775 [MediaServerConnection@217ee] MMC received response from 192.168.0.65:7008 of 78960
Tue 11/3 19:09:48.512 [MediaServerConnection@217ee] MMC sent request to 192.168.0.65:7008 of GET_FILE_SIZE D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:48.606 [MediaServerConnection@217ee] MMC received response from 192.168.0.65:7008 of 789600
Tue 11/3 19:09:48.655 [MediaServerConnection@217ee] MMC sent request to 192.168.0.65:7008 of GET_FILE_SIZE D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:48.655 [MediaServerConnection@217ee] MMC received response from 192.168.0.65:7008 of 911988
Tue 11/3 19:09:48.668 [MediaServerConnection@217ee] MMC sent request to 192.168.0.65:7008 of GET_FILE_SIZE D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:48.668 [MediaServerConnection@217ee] MMC received response from 192.168.0.65:7008 of 918568
Tue 11/3 19:09:48.672 [PooledThread@217ee] MMC sent request to 192.168.0.65:7008 of GET_FILE_SIZE D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:48.672 [PooledThread@217ee] MMC received response from 192.168.0.65:7008 of 918568
Tue 11/3 19:09:48.672 [PooledThread@217ee] Doing pre-emptive file format detection on recording MediaFile....D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:48.672 [PooledThread@217ee] MMC sent request to 192.168.0.65:7008 of GET_FILE_SIZE D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:48.673 [PooledThread@217ee] MMC received response from 192.168.0.65:7008 of 918568
Tue 11/3 19:09:48.764 [PooledThread@217ee] MMC sent request to 192.168.0.65:7008 of GET_FILE_SIZE D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:48.764 [PooledThread@217ee] MMC received response from 192.168.0.65:7008 of 1034376
Tue 11/3 19:09:48.767 [PooledThread@217ee] Using internal format detector first for: D:\RiverMonstersUnderwaterNightmares-1806823-0.ts
Tue 11/3 19:09:59.570 [MediaServerConnection@efd42f] Detected remuxing format of:AV-INF|f=MPEG2-PS;br=20384000;[bf=vid;f=MPEG2-Video;fps=29.970030;fpsn=30000;fpsd=1001;ar=1.777778;arn=16;ard=9;w=1920;h=1080;lace=1;index=0;tag=e0;main=yes];[bf=aud;f=AC3;sr=48000;br=384000;ch=6;index=1;tag=bd-81010002;lang=eng ;main=yes]; formatObject=MPEG2-PS 0:00:00 20384 kbps [#0 Video[MPEG2-Video 29.97003 fps 1920x1080 16:9 interlaced id=e0]#1 Audio[AC3 48000 Hz 6 channels 384 kbps MAIN idx=1 id=bd-81010002 eng ]]
Tue 11/3 19:09:59.614 [ProgressiveDeleter@1ac2227] Saving properties file to C:\Program Files (x86)\SageTV\SageTV\Sage.properties
Tue 11/3 19:09:59.631 [ProgressiveDeleter@1ac2227] Done writing out the data to the properties file
Tue 11/3 19:09:59.632 [ProgressiveDeleter@1ac2227] Completed progressive deletion of: D:\Armageddon-1783346-0.ts

[jvl711]

Not completely conclusive, but modified my network encoder to check to send the actual file size if it has access to the file. This seems to have possibly fixed the issue. Not 100% sure yet. Still need to figure out a better way to get the file size.

I have noticed there are a few channels were the remuxer does not seem able to come up with the format. Specifically it seemed to have an issue with Travel channel. Even after many megabytes it did not return a format. Is the format parser different in the remuxer than sage? I can give an example file if needed.

Josh

[Narflex]

The FormatParser has a conditional in it to not detect the format if the file is zero sized...however, if it's currently being recorded and the getRecordedBytes call returns a non-zero value, then it proceeds still (there were weird issues on Windows we would see where the report of the file size from the OS would lag behind the actual amount of bytes in the file you could actually read, that's why we check both ways). That's what you were hitting before.

I don't know if the format parser in the remuxer is different than the one SageTV is normally using...but I think they are the same (I don't think Qian would have wrote that code twice). If you can post an example file that reproduces the issue, that'd be great.