Issue with SageTV 9/Linux Build/SMB/Network Encoders

[Jabroni]

Was wondering if anyone noticed that when using the network encoder on the Linux build, whenever your Recording path is a SMB path, it seems that SageTV sends the wrong path to the Network Encoder to save the file to:

ie output of OpenDCT output when trying to tune in to a channel:

22:15:34.565 [SageTVRequestHandler-92:SlingIzzi] ERROR SageTVRequestHandler - SageTV sent: 'BUFFER SlingIzzi HDMI|1031268620|943|16777216|/opt/sagetv/server/smb:/FREENAS/SageTV/SlingIzzion192168549000HDMI-0.mpgbuf|Great-H.264', Replied: 'ERROR Device Start Failed'

So obviosly opendct throws exception cuz the path of the file is invalid (opendct is running on a Windows PC). Am I doing something bad or its an unexpected code from witihin the sage linux build ? (Im using the docker version)

Thanks,
Jabroni

[Fuzzy]

What is the actual library locations and such on your docker server? I'm certain this is just a problem with how the linux build is translating smb shares to their mount points - it likely is not untranslating them when it needs to share this information with the outside world.

Maybe post up the Sage.properties file as well as a log of an attempted recording - might make it easier to locate who is doing what.

[Jabroni]

Heres more info on my setup

Docker was created with mapped paths:
-v /mnt/media/SageTV:/var/media \ (this path would be irrelevant since the recording path would be the smb path)
-v /docker/sagetv:/opt/sagetv \

Theres a smb share where I want to put the recording which is \\freenas\SageTV (that is accesible to SageTV Server / VM running the Network Encoder / SageTV Clients)

The Recording path share was added thru the SageTV Placeshifter, went to parent path, network browse to the server, selected folder.

Relevant info regarding the Recording paths on Sage.properties

linux/smb_mounts=//freenas/sagetv,/tmp/sagetv_shares/freenas/sagetv;
seeker/video_storage=/tmp/sagetv_shares/freenas/sagetv,20000000000,3;

Log of Sage Server w/ a working video playback thru a direct HDHomeRun (without network encoder)

Tue 2/14 22:25:10.797 [AsyncWatch@3c6203e4] MediaFile startSegment enc=HDHomeRun 101b9d43 Tuner 0 KECYDT MediaFile[id=135587 FA[A[135587,0,"The X-Files",0@0214.22:21,5256000,]] mask=V host=docker encodedBy=HDHomeRun 101b9d43 Tuner 0 Digital TV Tuner format=MPEG2-PS 0:00:00 0 kbps []]
Tue 2/14 22:25:10.797 [AsyncWatch@3c6203e4] MediaFile created subfile:/tmp/sagetv_shares/freenas/sagetv/HDHomeRun101b9d43Tuner0DigitalTVTuner-0.mpgbuf MediaFile[id=135587 FA[A[135587,0,"The X-Files",0@0214.22:21,5256000,]] mask=V host=docker encodedBy=HDHomeRun 101b9d43 Tuner 0 Digital TV Tuner format=MPEG2-PS 0:00:00 0 kbps []]

That same format seems its being sent thru the network encoder file path, which would make it invalid for the network encoder if its hosted on another pc/vm/docker.

Ill do some more testing tomorrow.. 2am and im falling sleep on the keyboard

[Fuzzy]

Really strange. I have no idea why the mediafile is returning the mangled path to the NetworkCaptureDevice in your first post, but is returning at least a proper localized path to the HDHomeRunCaptureDevice in the second post.

I'll have to play around a bit and see what is going on where. I purposefully forced openDCT to use the MediaServer when I built the OpenDCT docker container to avoid path issues, but I figured it was to just avoid having to map paths the same between the Sage docker and the OpenDCT docker. I did not anticipate that the actual network encoder architecture was broken when recording to a network share. Can you post this info up to an issue on github, so hopefully some other eyes might get a crack at it as well (I likely won't get to look at this very deeply for quite a while I'm guessing).

[EnterNoEscape]

I know what's going on there. When SageTV is using a path, it usually uses getCanonicalFile() to fully resolve it into a direct path. This method doesn't know that in SageTV smb:// is a Samba share, so it just prepends the current directory with the assumption that's where it should be. In Linux : is a valid directory character which is why this doesn't just throw an exception. The // is consolidated into / because it's the same thing when applied to directories.

There is a bigger question/problem here. If we want the smb:// to be passed to the network encoder, this will only be acceptable when either the network encoder will be using the media server or the network encoder is running on Windows. Doing this in Linux without the media server would involve the network encoder automatically mounting the desired share which I really don't think a network encoder should be doing. I have good reason to believe that if you used the media server, it wouldn't create the correct file at the correct location because the value you see in the encoder request is the same value that the media server will try to open when the network encoder connects to it.

[mechling-burgh]

This is also happening when you put a smb share in import directories in sage. I have a mybook with a bunch of movies on it for my kids I also map it to Sage so the kids can watch on the TV. The smb share works fine in the windows version of sage but in the Docker version it gives me a strange path. I have not posted anything about since I was still experimenting with the docker and had other things I was worried about first.

[Fuzzy]

Sage DOES have some level of tracking the mappings between NFS/SMB shares and local paths with the linux/smb_mounts and linux/nfs_mounts properties. It is my opinion that MediaFile should actually store the user chosen path (the UNC path, if present), and those mount translations be done in the backend when the file is actually being accessed. I do not know what all this touches though, so it may be a bigger issue to fix it up proper. This would then result in the proper UNC being passed to the encoders, as that is the official location of the recording.

[EnterNoEscape]

Originally Posted by Fuzzy

Sage DOES have some level of tracking the mappings between NFS/SMB shares and local paths with the linux/smb_mounts and linux/nfs_mounts properties. It is my opinion that MediaFile should actually store the user chosen path (the UNC path, if present), and those mount translations be done in the backend when the file is actually being accessed. I do not know what all this touches though, so it may be a bigger issue to fix it up proper. This would then result in the proper UNC being passed to the encoders, as that is the official location of the recording.

UNC is a Windows convention. Linux only mounts things. When you access smb:// SageTV has quietly mounted it somewhere for you. That mount location should be what SageTV gives the network encoder in my opinion. I really don't think network encoders should be doing path translations; that's what the media server is for.

[Jabroni]

Ok so for the Network Encoder on OpenDCT, I switched the consumer to opendct.consumer.MediaServerConsumerImpl and it seemed to have eased the issue since the Network Encoder doesnt write directly to the file. But for legacy

Still not sure if how sage handle SMB/UNC path is the correct one, maybe if we could allow the user choose which path it would need/pass (cuz it would break compatibility with Legacy Network Encoders). For example in one of my Network Encoders I use DVB4Sage, and that uses the file path that the network encoder passes on the START to start saving the file. In that case, the mount point on the linux server would be broken/useless if the encoder is running on another machine (unless it uses the mediaserver to stream back to the Sage Server that is)

[Fuzzy]

Originally Posted by EnterNoEscape

UNC is a Windows convention. Linux only mounts things. When you access smb:// SageTV has quietly mounted it somewhere for you. That mount location should be what SageTV gives the network encoder in my opinion. I really don't think network encoders should be doing path translations; that's what the media server is for.

Yes, UNC paths are a 'windows convention', but so is smb networking to begin with, and UNC is still the 'proper' way to represent an smb location. The entire basis behind a network encoder is to have it running on a different maching on the network. Yes, I know there are other uses of it now that use it locally, but that is not the original intent. A local mount point means nothing to any other computer on the network, but smb://Server/path/to/file or //Server/path/to/file. On linux, either would need to be mounted. On windows, the latter (UNC) will 'just work', and frankly, is what the windows server has passing to the encoders for over a decade.

[EnterNoEscape]

Originally Posted by Fuzzy

Yes, UNC paths are a 'windows convention', but so is smb networking to begin with, and UNC is still the 'proper' way to represent an smb location. The entire basis behind a network encoder is to have it running on a different maching on the network. Yes, I know there are other uses of it now that use it locally, but that is not the original intent. A local mount point means nothing to any other computer on the network, but smb://Server/path/to/file or //Server/path/to/file. On linux, either would need to be mounted. On windows, the latter (UNC) will 'just work', and frankly, is what the windows server has passing to the encoders for over a decade.

So by your logic, we want Linux to return paths like it's Windows to network encoders in case the network encoder is running on Windows. I guess that would be fine as long as if you connect with the media server, it converts everything for you and any future Linux network encoders would need to be aware of this possible request. I think we can solve more headaches by just using the media server for all future network encoders. It's the best way to guarantee that if SageTV has access to it, the network encoder has access to it too.

[Jabroni]

Originally Posted by EnterNoEscape

So by your logic, we want Linux to return paths like it's Windows to network encoders in case the network encoder is running on Windows. I guess that would be fine as long as if you connect with the media server, it converts everything for you and any future Linux network encoders would need to be aware of this possible request. I think we can solve more headaches by just using the media server for all future network encoders. It's the best way to guarantee that if SageTV has access to it, the network encoder has access to it too.

Or we could have another preference to the Sage.properties for the specific encoder that defines which path to announce on the network encoder communication, so it wont break the support for old network encoder, and still support new ones. I think since the linux user base has been so low till recently that linux has the 64bit advantage + docker over the win32 build, this kind of issue didnt arrise that much. And more with Dockers since the idea of it is having all external data, either defined by docker or smb.

[Fuzzy]

Originally Posted by EnterNoEscape

So by your logic, we want Linux to return paths like it's Windows to network encoders in case the network encoder is running on Windows. I guess that would be fine as long as if you connect with the media server, it converts everything for you and any future Linux network encoders would need to be aware of this possible request. I think we can solve more headaches by just using the media server for all future network encoders. It's the best way to guarantee that if SageTV has access to it, the network encoder has access to it too.

What I'm suggesting is that in the network encoder protocol should require some standard method of passing network paths. That is all. UNC IS a standard, that IS recognized in the unix world as well (just with forward slashes instead of backslashes). The UNC path is what is used in the mount command, NOT smb://.

My problem here has more to do with how Sage deals with UNC pathed folders in general. They should always be referred to as their UNC path, because that is what the user entered in when adding the folder. The conversions being done on the addfolder methods is, in my opinion, the wrong way - and the translation to a mount point should instead be done when the file is accessed.

Currently, if you are adding a share to a linux server, here's what happens:
Add New Directoty -> Browse up to root and choose Network -> browse through network through workgroup, server, folder, etc -> choose the path and the path listed that you've browsed to is (as an example):
smb://TOWER/music
Then, when you select that folder to be added, it gets added to the list of Library Import Directories as
/tmp/sagetv_shares/tower/music
THIS is the first problem, and really needs to be fixed there.

Alternatively, if you choose Specify... and enter ANY valid method of relating to a samba share, none of them work. smb://TOWER/music, //TOWER/music, \\TOWER\music, etc all return "The directory<path> does not exist, Would you like to create it?"
THIS is the second problem.

[EnterNoEscape]

Originally Posted by Fuzzy

What I'm suggesting is that in the network encoder protocol should require some standard method of passing network paths. That is all. UNC IS a standard, that IS recognized in the unix world as well (just with forward slashes instead of backslashes). The UNC path is what is used in the mount command, NOT smb://.

My problem here has more to do with how Sage deals with UNC pathed folders in general. They should always be referred to as their UNC path, because that is what the user entered in when adding the folder. The conversions being done on the addfolder methods is, in my opinion, the wrong way - and the translation to a mount point should instead be done when the file is accessed.

Currently, if you are adding a share to a linux server, here's what happens:
Add New Directoty -> Browse up to root and choose Network -> browse through network through workgroup, server, folder, etc -> choose the path and the path listed that you've browsed to is (as an example):
smb://TOWER/music
Then, when you select that folder to be added, it gets added to the list of Library Import Directories as
/tmp/sagetv_shares/tower/music
THIS is the first problem, and really needs to be fixed there.

Alternatively, if you choose Specify... and enter ANY valid method of relating to a samba share, none of them work. smb://TOWER/music, //TOWER/music, \\TOWER\music, etc all return "The directory<path> does not exist, Would you like to create it?"
THIS is the second problem.

I am actually playing in this particular code right now and I agree that the second issue is a fairly big issue and frustrated me a lot when I was playing around with SageTV on Linux for the first time. I could take a stab at it, but just know that if I fix it, it will likely only be a part of Library (a class I separated these things into) and you will only be able to use the fixed approach by turning on a beta feature.

[stuckless]

Originally Posted by Fuzzy

What I'm suggesting is that in the network encoder protocol should require some standard method of passing network paths. That is all. UNC IS a standard, that IS recognized in the unix world as well (just with forward slashes instead of backslashes). The UNC path is what is used in the mount command, NOT smb://.

smb:// is standard as well... but the problem with standards is that there so many to choose from.

protocol://server/path is universal URI standard for referencing resources, whether they be local or over a network.

smb://server/path is perfectly valid on a system that has a URI handler registered for 'smb'... like wise file:///path is a valid uri to reference a file.

I think internally (from some of the code that I've seen), sagetv doesn't really do a good job of handling these URIs even though it does try to handle them in some cases (just not consistent).

Windows might understand file:, ftp:, smb:, etc, not sure (but I'd be surprised if it didn't understand it)... They are certainly understood in java (if the protocol handler is registered) and the unix system as a whole.

[Fuzzy]

The inconsistency is the problem. The one thing sage has, in the past, usually worked well with was UNC paths - that's the main reason I think we should try to 'standardize' towards that. The UNC path is how the linux version is writing the pairing in the smb_mounts property as well. It's just a few places that they don't work, and they are ultimately forgotten about with the libraries, which I can certainly see as confusing to some users when they only added a single library folder, and their files are all showing that they are in some temp directory.