Feature request: Scan video libraries only

[wayner]

In my Sage libraries I probably have tens of thousands of songs and tens of thousands of photos (I use the photo screensaver). I think I have maybe a couple of thousand of video shows - lots of kids' TV shows plus movies.

When I do a media library scan it takes something like 5-6 minutes. A few months ago I set Sage up on another server and I only had the video files installed and I was surprised to see that my media scans took a matter of seconds. This must mean the scans on my main server are bogged down by the photo and music files.

It would be great if you could only scan the video libraries (or scan separately for music/photos/videos/all) as that would make it much easier to get a new movie/show up in Sage by saving me 5 minutes of scan time. I doubt that I am the only one with this issue.

[stuckless]

Think this would be an awesome feature... sort of lines up with the post I had a couple weeks ago, when I said that it would be possible to actually do a real import of media files as they are added. ie, Java 7 has the ability to "watch" the filesystem for changes, so it would be possible to notify sagetv to load a single video when it is added, instead of having it do a full scan.

[Monedeath]

http://forums.sagetv.com/forums/show...0&postcount=48

Is the post/thread in question.

Yes it would be a nice feature to have in place.

Although staying generally on topic, I know someone who would like to split the pictures and music DB off from the wiz.bin entirely. So it would conceivably fall in the scope of that kind of project. But he's still busy chasing other non-Sage projects right now, and just starting to dip his toes into the proverbial coding waters involving Sage. I don't think that kind of undertaking would be something that could done without a LOT of work(weeks if not months worth), and I doubt there would be much gained. Thankfully I think it's low on his list of things he'd like to do to Sage once he does work his way over here.

[Narflex]

SageTV does sleep periodically when it's doing library import scans so as to not overload the filesystem with requests and interfere with playback/recording from/to the disk. This is done in the sage.Seeker.controlImportCPUUsage() method. If you want to put a property in there which allows disabling that CPU throttling; go for it, I'll approve it.

[stuckless]

Originally Posted by Narflex

SageTV does sleep periodically when it's doing library import scans so as to not overload the filesystem with requests and interfere with playback/recording from/to the disk. This is done in the sage.Seeker.controlImportCPUUsage() method. If you want to put a property in there which allows disabling that CPU throttling; go for it, I'll approve it.

HOw much effort (in your mind) would it be to notify SageTV to just scan an import a single file? (ie, if we had a process that notified sagetv of a new single file.. it can scan and load it, instead of doing a full scan)

[Monedeath]

Originally Posted by stuckless

HOw much effort (in your mind) would it be to notify SageTV to just scan an import a single file? (ie, if we had a process that notified sagetv of a new single file.. it can scan and load it, instead of doing a full scan)

Or for that matter, a single import directory.

[Fuzzy]

Originally Posted by stuckless

HOw much effort (in your mind) would it be to notify SageTV to just scan an import a single file? (ie, if we had a process that notified sagetv of a new single file.. it can scan and load it, instead of doing a full scan)

Well, I think there are calls to directly add an item to the database, but I'm not sure if this process itself will trigger 3rd party metadata searches.

[stanger89]

I still really like the idea of just monitoring the filesystem to get notified of file additions, rather than having to do rescans.

[stuckless]

Originally Posted by stanger89

I still really like the idea of just monitoring the filesystem to get notified of file additions, rather than having to do rescans.

I have looked at this very briefly, and while we could just do an Add like Fuzzy stated, I'd rather tell sagetv to do a normal scan with a "hint" where the hint is folder or file to limit the scope, and then have it do it's normal thing. The seeker code doesn't seem to be setup that way, from what I can tell, (again very quick look on my part), but, that's not to say we couldn't change it. The monitoring part is pretty easy (java 7).

[Monedeath]

http://sagetv.com/api/sage/api/Media...a.lang.String)

Seems to be geared toward a targeted single file import. This probably is the API call mentioned previously.

Digging further it looks like "java/sage/MediaFile.java" is handling the import process, but at my (basic) level of knowledge, I'm going to bow out of the chase from here.

[PLUCKYHD]

Originally Posted by stanger89

I still really like the idea of just monitoring the filesystem to get notified of file additions, rather than having to do rescans.

I've started with this using Java 7 api Stuckless posted it's pretty straightforward but for now is seems to be better fit as plugin vs core because of Java ver requirement

[stanger89]

Couldn't Sage 9 just require Java 7 or greater? Is there any reason to maintain backward compatibility to Java 6?

[Monedeath]

Originally Posted by stanger89

Couldn't Sage 9 just require Java 7 or greater? Is there any reason to maintain backward compatibility to Java 6?

Narflex specified Java 1.5 as the current minimum version when Sage went OS.

Putting conditional code into the core to take advantage of Java 1.7+ has been discussed, but consensus was that implementing as a plugin probably was a better route to pursue to keep the code in core from getting crazy with conditionals. I don't recall Narflex weighing into things one way or the other.

[stanger89]

The release notes said "Java compliance level is now 1.5", but I've never read anything saying there was a desire to keep it there. It was mainly of note because the JVM on the extenders was 1.4 so extender code can't be built from the OS code.

[Narflex]

I still use Java 1.6 at home...but it shouldn't be that hard for me to upgrade that (I have a unique system that's pretty difficult to update outside of the SageTV stuff generally though).

And I don't think this is a big problem to solve either. If the issue is how long it takes; that can easily be sped up by not doing the throttling under manual request.

[Fuzzy]

Originally Posted by Monedeath

http://sagetv.com/api/sage/api/Media...a.lang.String)

Seems to be geared toward a targeted single file import. This probably is the API call mentioned previously.

Digging further it looks like "java/sage/MediaFile.java" is handling the import process, but at my (basic) level of knowledge, I'm going to bow out of the chase from here.

Yes, that would add a file to the wiz.bin, but it doesn't trigger any events when done, so phoenix would not run on the file, for instance. If we want to do a targetted scan, it might not be too difficult to make an overload for seeker's process to limit the search to a particular part, there are a few hangups though. The first thing it does is make a list of all files from the scan, and then it parses through them and handles each one (checking if it is in the database already, adding it if it is not, etc). That part would be easy to simply limit that list generation. However, there's also logic to remove files form the database that are NOT there anymore, and that is where things would get more complicated.

Perhaps something that may be even easier to speed up new additions, is when the list is generated, have it sorted by the file's timestamp, descending (currently, it's sorted alphabetically). This would get the newest additions first, and maybe get your new files in there quicker. There needs to still be some alphabetizing, however, as it is currently sorted alphabetically to ensure -0 files are processed before -1, -2, etc. on multisegment recordings.