ATT Stanger89 - Help needed

[stanger89]

Originally Posted by mikesm

Gerry, Stanger, thanks for the replies.

Removing the videos directory, rescanning, and then re-adding the video directory and rescanning again doesn't remove the meta-data. After doing this, I still get the same titles with meta-data in there as before.

There seems to be something odd about the way Sage stores this stuff. Look at this thread: http://forums.sagetv.com/forums/show...media+database
This looks like the same problem I have. The meta-data hangs around even if the files go away.

I thought the database was stored on the server, so even if I did this on the clients only, the actual database modifications were being made centrally so that all the other clients would see the changes.

Yup, exactly right.

I will try and install this on the server itself if you think I should, but I really try and leave that system alone... :-)

My server is headless, like I said I always try these things on clients, I've never tried it on the server.

I don't see any error messages displayed about titles that didn't import correctly. Is there a log file where problems are recorded somewhere?

If you have logging enabled, you can try looking in the client's log file, though I'm not sure if SageMC logs anything.

[mikesm]

Originally Posted by flachbar

Try this :

instead of simply removing the import directories, rename them and create new empty directories with the same name in the same location, so that Sage sees the existing import directories as empty. Then do a refresh, and wait until the videos are gone in your library. Then rename your real directories back, and do another refresh. Works for me all the time.

An no, there's really no need to do this on the server...


Dirk

Ugh, this stuff keeps surprising me with what happens. Ok, I did what you suggested, let me tell you what happened:

1) All my video is stored on the server in a exported filesystem called Videos. In there are 4 directories, DVD (where the DVD rips are stored in 1 directory per DVD), Internet (video files downloaded from Internet web sites, in directories by genre (Documentary, webcasts, etc..), Replay_Video (where Dvarchive stores videos downloaded from Replay TV's), and SageTV (where Sage records it's programming from tuners).

2) Sage had been configured to import videos from the Video directory. It never picked up SageTV recordings as importable because it's the Sage recording dircetory). Each folder (DVD, Internet, etc...) showed up seperate in the filesystem view).

3) I renamed Video as Video1, made a new empty dir called Videos, and rescanned. Nothing showed up in the My Videos. I then removed Video, and renamed Video1 as Videos, and then did a rescan.

4) The same DVD's that had had meta data for them before showed up exactly the same way. No data was cleared. Doing the import etc... of dvdprofiler info resulted in the smae problem as before. No change.

Additionally, several of the programs that Sage recorded recently also showed up. I didn't think this would be the case as it never happened before.

5) So now I reconfigure the video import directories list to import only DVD, Internet, and Replay_video as 3 separate imports, leaving out sage, and deleting the root Videos import. I do a rescan.

6) The DVD problem is still there as before. The imported videos from SageTV directory STILL show up, even though that dir is no longer in the import list. I don't particularly want to do much messing around with renaming that directory, because Sage records to it, and I don't wnat to lose the metadata for recorded programming. The funny thing is that there are hundreds of recorded videos in that directory, but only about 12 of them got imported in the first place. How do I get rid of these appearing in the My Videos directory?

Additionally, I never do a background rescan. I wait until the rescan is done before doing anything, so I don't think it's the case that I didn't wait long enough for the rescan to work.

These problems sound a like like teh problems in that thread I referenced earlier about Sage holding on to metadata in improper ways...


Thanks,
Mike

[mikesm]

One more thought...

Even without changes to the GUI, it seems to me that it might be pretty easy to hook My Movies 2's database directly into Sage. My Movies 2 stores it's database in an open microsoft access format. You use the collection manager standalone application to set up the database like dvdprofiler, but unlike dvdprofiler, the paths to the actual ripped files are stored in there.

Given this, isn't it straightforward to insert these entries from the access database file into the Sage TV media database, file path and all? No scanning needed, no need to make sure directories have the exact same name, etc...

This seems like an extension of the DVDPro2Sage addin, but a little more straightforward because you have the filesystem location already configured.

Thanks,
Mike

[stanger89]

The "problem" is MyMovies works kind of the opposite way Sage+DVD Profiler does, what I mean is MM use the MM database to store a list of DVDs, and goes directly to it for all it's information. Without the MM database, MCE doesn't know about any of the movies.

In Sage, Sage has it's own internal database, and knows about all your movies with or without an external database, the external database is used to lookup the data/coverart for movies Sage already knows about. You still need a way for Sage to identify which titles in the external database are associated with which movies on the HDD. The only way to do that that I know of, is by comparing titles.

This seems like an extension of the DVDPro2Sage addin, but a little more straightforward because you have the filesystem location already configured.

Ignoring for the moment that I have no idea how to interface with access from java (XML is pretty easy), you still need to figure out what title to lookup.

Also, Dirk added something to DVDPro2Sage to store/retrieve the "folder" that a movie belongs in, I'm not entirely sure how that all integrates with SageMC and everything. I think he was using one of the fields in DVD Profiler to store the path, but I'm not sure.

[dflachbart]

Originally Posted by stanger89

Also, Dirk added something to DVDPro2Sage to store/retrieve the "folder" that a movie belongs in, I'm not entirely sure how that all integrates with SageMC and everything. I think he was using one of the fields in DVD Profiler to store the path, but I'm not sure.

Yeah, this allows to store the video files wherever you want, independent of the actual video title. To tell Sage where to look for the DVD, you have to enter the full path into the attribute "OriginalTitle". This works transparently, so whenever DVDPro2Sage doesn't find the DVD in the "DVD title" folder, it will also look for the file in the folder specified by 'OriginalTitle'...


Dirk

[stanger89]

That's only in SageMC now.

[mikesm]

Originally Posted by stanger89

The "problem" is MyMovies works kind of the opposite way Sage+DVD Profiler does, what I mean is MM use the MM database to store a list of DVDs, and goes directly to it for all it's information. Without the MM database, MCE doesn't know about any of the movies.

In Sage, Sage has it's own internal database, and knows about all your movies with or without an external database, the external database is used to lookup the data/coverart for movies Sage already knows about. You still need a way for Sage to identify which titles in the external database are associated with which movies on the HDD. The only way to do that that I know of, is by comparing titles.



Ignoring for the moment that I have no idea how to interface with access from java (XML is pretty easy), you still need to figure out what title to lookup.

Also, Dirk added something to DVDPro2Sage to store/retrieve the "folder" that a movie belongs in, I'm not entirely sure how that all integrates with SageMC and everything. I think he was using one of the fields in DVD Profiler to store the path, but I'm not sure.

Stanger, sorry, I wasn't suggesting that Sage try and use the MM access database for movies, and it's own for everything else, rather, use it to "sync" with Sage's database in the way you do today with DVDprofiler, but in a more complete way.

I think this could be accomplished in a way that works in sageMC and the default STV, but reading from the MM access file and loading the DVD metadata and target file locations into the Sage database, without any "importing" of directories. Once loaded, if deleted from MM, it would be deleted from Sage's database too, or altered if the MM entry were altered.

This would also allow the Sage data to be deleted without jumping through circles if bad data got loaded into it. The syncing is done today through DVD profiler in a fashion that needs a lot of help, using fake .mpg files and rescanning of files, etc... This would seem to be cleaner to me, since MM stores all the appropriate data. And has a standard way to represent musti-disc boxsets, though that would probably take some UI work to integrate properly.

As for linking access to java, there are a bunch of references to using jdbc-odbc bridges to do this. I'm no java programmer, but it seems there are quite a few libraries out there to do that.

MM seems a lot easier to use than DVD profiler, esp since the conversion to 3.0 broke so much stuff. And it's free on top of it all.

What am I missing here?

Thanks
mike

[stanger89]

Originally Posted by mikesm

Stanger, sorry, I wasn't suggesting that Sage try and use the MM access database for movies, and it's own for everything else, rather, use it to "sync" with Sage's database in the way you do today with DVDprofiler, but in a more complete way.

But that's exactly what you're suggesting, if you don't import directories, then you're basically using two separate databases. I really don't know how Sage handled files that aren't in one of it's import or recording paths.

This would also allow the Sage data to be deleted without jumping through circles if bad data got loaded into it. The syncing is done today through DVD profiler in a fashion that needs a lot of help, using fake .mpg files

That's just for Offline media.

and rescanning of files, etc... This would seem to be cleaner to me, since MM stores all the appropriate data. And has a standard way to represent musti-disc boxsets, though that would probably take some UI work to integrate properly.

DVD Profiler has a "standard way" of doing box sets to, it's the UI work that's yet to be done on that front.

As for linking access to java, there are a bunch of references to using jdbc-odbc bridges to do this. I'm no java programmer, but it seems there are quite a few libraries out there to do that.

MM seems a lot easier to use than DVD profiler, esp since the conversion to 3.0 broke so much stuff. And it's free on top of it all.

What am I missing here?

That's a lot of work for no payoff (for me). I don't have paths in a database, and it would take me a very, very long time to enter paths for all my (almost 400) DVDs into MM.

Again, it's not the MM database that makes it desireable, it's the MM UI, switching to using the MM database as the data source would leave us in exactly the same situation we have now, just with a different data source.

[KJake]

I looked around on SF.net, but didn't find any projects for this. Is the posted 05-26-2007 the latest working version? Thanks for your work on this!

-Kevin

[stanger89]

Probably, I think this is the latest one:
http://forums.sagetv.com/forums/show...3&postcount=78

[KJake]

Thanks, that one is newer indeed. I still have the same problem though, and it is probably more of a SageMC problem, I'll ping flachbar about it.

Basically, the DVDProfiler Edition tag is imported as Episode and when Episode is imported (exists), the description isn't shown in SageMC. Pretty odd.

[stanger89]

I thought SageMC came with the "right" version of DVDPro2Sage.

FWIW, DVDPro2Sage is "dumb" it basically just pulls the field requested out of the XML. It's up to the STV to put it in the right place in Sage.

[dflachbart]

Originally Posted by KJake

Thanks, that one is newer indeed. I still have the same problem though, and it is probably more of a SageMC problem, I'll ping flachbar about it.

Basically, the DVDProfiler Edition tag is imported as Episode and when Episode is imported (exists), the description isn't shown in SageMC. Pretty odd.

Hi,

I just verified on my machine that the version Stanger linked to as well as the one included in the full SageMC download zip both successfully import the DVD Profiler data.

What "Episode" and "Edition" tags are you referring to ? Stangers DVD2Profiler code does not read those tags (actually they don't even exist in my collection.xml file). The SageMC import code maps the DVD Profiler "Title" tag to the SageTV "Episode" field, and the "Overview" tag to the "Description" field. This has always been the case and never changed (except the necessary title/episode field swap due to changes in SageTV6).

Or are you talking about DVD Profiler 3.0 data ? Since I don't use it, I have never tested SageMC with the newer version... Though I always assumed that it should still work if you use Stangers newest class file, provided that his interface returns the same values for 3.0 as for 2.4.0.

Dirk

[KJake]

Originally Posted by flachbar

Hi,

I just verified on my machine that the version Stanger linked to as well as the one included in the full SageMC download zip both successfully import the DVD Profiler data.

What "Episode" and "Edition" tags are you referring to ? Stangers DVD2Profiler code does not read those tags (actually they don't even exist in my collection.xml file). The SageMC import code maps the DVD Profiler "Title" tag to the SageTV "Episode" field, and the "Overview" tag to the "Description" field. This has always been the case and never changed (except the necessary title/episode field swap due to changes in SageTV6).

Or are you talking about DVD Profiler 3.0 data ? Since I don't use it, I have never tested SageMC with the newer version... Though I always assumed that it should still work if you use Stangers newest class file, provided that his interface returns the same values for 3.0 as for 2.4.0.

Dirk

Exactly right, I'm using 3.0.3 - just bought it for this purpose. The Edition field in DVD Profiler is coming in as the Episode title in SageMC and when that happens, the Description isn't shown.

The Edition field in DVD Profiler is used for things like "Full Screen" or "Widescreen" or in the case of my James Bond DVDs, they're filled in as "Ultimate". On these DVDs, I don't get to see the description.

If I go to the Edit Video Info in SageMC and choose DVD Profiler Import, I can see that it has imported all the information correctly, just for some reason it won't display it all in the main title screen. I'm using SageMC 6.3.3b and Stanger's latest class file.

Thanks for monitoring this thread too Dirk!

[bcjenkins]

I too can confirm the issue, haven't had time to dig into it though. (Thanks Kevin)

[bcjenkins]

Another issue discovered with 6.2.5; upon importing IVL .7.2 the program guide selection bar seems to be changed. Screen attached. Thanks!

B

[KJake]

Hey Mike, didn't mean to hijack this thread

I had a problem where Sage was holding on to my files and these property settings finally made it clean itself up so that I could reimport my music and video directories.

Try it at your own risk...

Remove your imported directories, shutdown sage server process/service, make a backup of the Wiz.bin, place these in the Sage.properties and start the sage server back up.

make_all_files_local=true
make_all_mediafiles_local=true
force_full_content_reindex=true

I would have debug logging on while you do this and check the log file to see if it says anything that looks like it is cleaning up the database. In my case, since it was holding on to files that weren't there any more, it finally started removing them from the DB.

I know these two things sound a little unrelated, but if you haven't tried this already, it can't hurt (that I know of ).

I wish we had a little more control over the core database. I even suggested in my message to support that Wiz.bin should be only for core-Sage stuff and that it would be nice to have multiple satelite files for things like .my meta-data...that way if stuff screws up, delete the add-on DBs and start over with those -- not all of Sage.

[edit]By the way, my server and sage recordings folder isn't on the same computer and these settings didn't effect them at all, they only seemed to care about my imports...who knows.[/edit]

[mikesm]

Interesting. This doesn't clean the meta-data off Sage recordings right, just stuff in the import directories?

I've never heard of these properties before. What does setting these proprties do? That is, what does sage do differently if make_all_files_local = true?

Thanks,
Mike

[Opus4]

Originally Posted by mikesm

I've never heard of these properties before. What does setting these proprties do? That is, what does sage do differently if make_all_files_local = true?

Nothing; that isn't a property that SageTV uses, as far as I know, unless I just don't have it in my properties file. It used to use make_all_mediafiles_local=true to note that the files are no longer on the machine w/the network name where they used to be so that the files could be recognized on a PC w/a different network name. Now, that property is always true and I don't believe it does much of anything at all.

If you have force_full_content_reindex=true when SageTV starts, that will force it to reimport everything in your library. If you have customized any metadata via any of the various plugins for that, all that data will be dumped since the files will now be freshly imported.

- Andy

[mikesm]

Originally Posted by Opus4

Nothing; that isn't a property that SageTV uses, as far as I know, unless I just don't have it in my properties file. It used to use make_all_mediafiles_local=true to note that the files are no longer on the machine w/the network name where they used to be so that the files could be recognized on a PC w/a different network name. Now, that property is always true and I don't believe it does much of anything at all.

If you have force_full_content_reindex=true when SageTV starts, that will force it to reimport everything in your library. If you have customized any metadata via any of the various plugins for that, all that data will be dumped since the files will now be freshly imported.

- Andy

Does force_full_content_reindex affect SageTV recordings, or only imported videos? I have a bunch of older SageTV recordings that would not have a valid guide ID to generate meta-data, so I want to keep all that, but completely purge the imported video data and start over for that. Will this property do that?

Thanks,
Mike