Wiz.bin alternative?

[GTwannabe]

Over they years, my Wiz.bin file has become progressively more bloated. As it grows, it takes considerably longer to start the SageTV service.

Does Sage plan to offer an alternative to the Wiz.bin internal database? A MySQL or SQL Server Express option perhaps?

[ToxMox]

I crave this as well.

[GKusnick]

I'm curious what you would use this for that you can't accomplish through the existing SageTV API, which gives pretty much complete access to the contents of wiz.bin.

[stevech]

Originally Posted by GTwannabe

Over they years, my Wiz.bin file has become progressively more bloated. As it grows, it takes considerably longer to start the SageTV service.

Does Sage plan to offer an alternative to the Wiz.bin internal database? A MySQL or SQL Server Express option perhaps?

Would much of that be the database functions that track "Watched" ? Not obvious that a different DBMS would make a big difference.

[IVB]

how often do you load the service that this is an issue?

I only restart that machine every, well, i don't actually know. I may have restarted it 6 months ago, i'm not quite sure,

[nielm]

Originally Posted by GTwannabe

Over they years, my Wiz.bin file has become progressively more bloated. As it grows, it takes considerably longer to start the SageTV service.

Define 'bloat'.

Wiz.bin grows because it remembers all the programs you have recorded and the ones you have watched, and the ones you have marked as don't like.

It needs to remember these shows to function correctly.

If you *really* want to clean up the wiz.bin, you can use my webserver to find all watched/dont like airings in the DB and clear the the watched/dont like status. At the next restart or EPG update, they should be removed. (but see below)

You can also do this by shutting down Sage, setting the following properties, then restarting it (backup wiz.bin first)

wizard/clearprofile=true
wizard/clearwatched=true

Note that If you use IR, this will completely upset the IR profiles as it will no longer know which shows you have watched (like) and don't like... It will also re-record watched and deleted files.

In 6.1 it also remembers the show info for all recorded files - even if they no longer exist - so that you can move them around.
You need to shutdown and set this property to clear the show information for shows which no longer have recordings:

wizard/retain_airings_from_completed_recordings=false

It also knows about all your imported pics, music and vids, so as you add new ones, the size will of course increase.

Sage is unlikely to replace it's database backend. The system is currently fast and reliable having been refined over several years, and a change will have no benefit for 99% of users...

[dvd_maniac]

It also knows about all your imported pics, music and vids, so as you add new ones, the size will of course increase.

I started with a wiz.bin of 32.9MB
I downloaded the WebServer and deleted all watched shows I no longer needed. It went from almost 6000 watched shows down to 126.
I then shutdown Sage and Service and restarted, left it for an hour and came back to see the wiz.bin was 33.4MB???

I then removed all my imports (over 5500+), shutdown Sage and Service again and restarted, The wiz.bin then shrunk to 18.1MB...

Then I reimported all my import videos again and after an hour the wiz.bin is at 40.6MB... &^$%&$%

Does Sage remember the watched status of imported videos even after they are removed from imported videos?

What else would cause such a jump in size and why didn't deleting over 5000 watched shows have any impact on the filesize of the wiz.bin?

[ybrew]

Originally Posted by dvd_maniac

I started with a wiz.bin of 32.9MB
I downloaded the WebServer and deleted all watched shows I no longer needed. It went from almost 6000 watched shows down to 126.
I then shutdown Sage and Service and restarted, left it for an hour and came back to see the wiz.bin was 33.4MB???

I then removed all my imports (over 5500+), shutdown Sage and Service again and restarted, The wiz.bin then shrunk to 18.1MB...

Then I reimported all my import videos again and after an hour the wiz.bin is at 40.6MB... &^$%&$%

Does Sage remember the watched status of imported videos even after they are removed from imported videos?

What else would cause such a jump in size and why didn't deleting over 5000 watched shows have any impact on the filesize of the wiz.bin?

I'm making an assumption here, but I'm assuming the wiz.bin didn't shrink because a compact process needs to be called to do that. Your wiz.bin file is larger, but it probably contains a lot more empty pages. It should be much faster to load, but until it's compacted and the empty/white pages are deleted, it will continue to consume more disk space.

[CadErik]

Originally Posted by dvd_maniac

I started with a wiz.bin of 32.9MB
I downloaded the WebServer and deleted all watched shows I no longer needed. It went from almost 6000 watched shows down to 126.
I then shutdown Sage and Service and restarted, left it for an hour and came back to see the wiz.bin was 33.4MB???

I then removed all my imports (over 5500+), shutdown Sage and Service again and restarted, The wiz.bin then shrunk to 18.1MB...

Then I reimported all my import videos again and after an hour the wiz.bin is at 40.6MB... &^$%&$%

If this is properly implemented (with an index and if sage doesn't need to always load everything in memory), the size of your wiz.bin shouldn't matter at all. Otherwise, yes a DBMS would make sense.

Erik.

[nielm]

Originally Posted by dvd_maniac

I started with a wiz.bin of 32.9MB
I downloaded the WebServer and deleted all watched shows I no longer needed. It went from almost 6000 watched shows down to 126.
I then shutdown Sage and Service and restarted, left it for an hour and came back to see the wiz.bin was 33.4MB???
...
why didn't deleting over 5000 watched shows have any impact on the filesize of the wiz.bin?

You didn't delete them, you cleared the watched flag...
If you are using 6.1 or above you need to set the 'wizard/retain...' property mentioned above to remove airings for previously-recorded-and-deleted files, and even then they may not get removed until a server restart or an EPG update (previous posted edited to mention this)

Does Sage remember the watched status of imported videos even after they are removed from imported videos?

don't think so... When an impvid is removed, it's Airing object is deleted.

[GTwannabe]

Originally Posted by nielm

Define 'bloat'.

Wiz.bin grows because it remembers all the programs you have recorded and the ones you have watched, and the ones you have marked as don't like.

It needs to remember these shows to function correctly.

If you *really* want to clean up the wiz.bin, you can use my webserver to find all watched/dont like airings in the DB and clear the the watched/dont like status. At the next restart or EPG update, they should be removed. (but see below)

You can also do this by shutting down Sage, setting the following properties, then restarting it (backup wiz.bin first)

wizard/clearprofile=true
wizard/clearwatched=true

Note that If you use IR, this will completely upset the IR profiles as it will no longer know which shows you have watched (like) and don't like... It will also re-record watched and deleted files.

In 6.1 it also remembers the show info for all recorded files - even if they no longer exist - so that you can move them around.
You need to shutdown and set this property to clear the show information for shows which no longer have recordings:

wizard/retain_airings_from_completed_recordings=false

It also knows about all your imported pics, music and vids, so as you add new ones, the size will of course increase.

Sage is unlikely to replace it's database backend. The system is currently fast and reliable having been refined over several years, and a change will have no benefit for 99% of users...

When SageTV loads, it pegs one CPU for ~5 minutes and won't accept connections from my Clients. My server is no slouch (3ghz, hyperthreading, 2GB RAM). I can only assume that Sage is slogging through my rather large Wiz.bin inefficiently. Utilizing a multithreaded, caching database engine would certainly be faster and more robust.

While I could purge my Wiz.bin and nuke my 10,000+ imported media files, that really defeats the purpose of SageTV.

[dvd_maniac]

hmm. the wiz.bin was still at 29.6MB even after changing the wizard/retain to false, restarting server and forcing an EPG update.

Then I removed my music folder import directory...
21.1MB

I have about 15,000 songs in there. I will have to go through them and trim the fat, so to speak. I will also see if this helps with menu response time for a week or so before I put them back in.

[GKusnick]

Originally Posted by GTwannabe

Utilizing a multithreaded, caching database engine would certainly be faster and more robust.

Using an off-the-shelf database engine might have been faster and more robust if they'd done it that way from the beginning. It does not follow that switching out database engines now that the product is mature would improve performance or reliability; quite the opposite, I expect.

Also it's not clear that the current engine doesn't cache. It may just be that if you have IR enabled, it has to touch a whole lot of entries right off the bat in order to decide what to record. Caching won't help you much there.

But this is really all speculation since none of us really knows what's going on during database startup. So it seems like a stretch to say that a different database engine would certainly be better.

[dvd_maniac]

Removed all my Imported Videos so the DB has a clean watched history, no Imports to scan and a very small list of upcoming recordings since it is off-season.
Results = 10.3MB wiz.bin and it loads SageService and Sage in just around 1 minute.
It used to take me over 4 minutes to completedly load before...
Also, I've been playing around the Menus and have not noticed any of the slow responsiveness I've become accustommed to.


Obviously the wiz.bin is here to stay but I wonder if there is a limit to how big it can get before it makes Sage unstable?

Also, why does the imported videos make the menu responsiveness slow even when not in the Imported Video screen?

[GTwannabe]

Originally Posted by dvd_maniac

Removed all my Imported Videos so the DB has a clean watched history, no Imports to scan and a very small list of upcoming recordings since it is off-season.
Results = 10.3MB wiz.bin and it loads SageService and Sage in just around 1 minute.
It used to take me over 4 minutes to completedly load before...
Also, I've been playing around the Menus and have not noticed any of the slow responsiveness I've become accustommed to.


Obviously the wiz.bin is here to stay but I wonder if there is a limit to how big it can get before it makes Sage unstable?

Also, why does the imported videos make the menu responsiveness slow even when not in the Imported Video screen?

This illustrates why I believe Sage should look at other database engines. A 2x increase in file size should not result in a 4x increase in startup time. Sage specializes in PVR software, not writing high performance data storage engines.

A 20mb Wiz.bin should be no problem for any modern database engine. Of course, some very functional, yet unoptimized engines (Access) will get the job done, albeit slowly. Query a 20mb Access database and you'll be waiting a long time. Query the same 20mb worth of tables in MySQL or SQL Server and you'll have your results in seconds.

[GKusnick]

I agree that N-squared growth in startup time isn't good, but what's your basis for concluding that the DB engine must be to blame for it? For all we know there could be some higher-level process, related perhaps to the recording scheduler or EPG update or media library maintenance, with an N-squared component to it. Swapping out DB engines wouldn't help in that case.

[dvd_maniac]

I agree that a SQL database would be of benefit. However, like previously posted, the wiz.bin is already intergrated into Sage and has matured greatly in the past few years. As it stands now the majority of people do not have a multitude of imported videos, music and/or do not do so much recordings that the wiz.bin growes so big.

Instead of saying that Sage should change it now(which we know is never going to happen), how about some work around ideas like having 3 databases? One for recordings, Favorites and epg, another for Imported Videos and the last for Music & Pictures...
This way it will only query the database you request and performance shouldn't be degraded for those with mediuim to large depository.

As for other things affecting my startup times? I tested with and without my Imported Videos/Music more than 10x and the results were around the same everytime. Even my bootup time improved without them.

[wvpolekat]

DB engines and such aside, the single biggest problem sage has with todays average computer is that it is single threaded. 2 CPUs/cores don't help it one little bit. It does help in that other processes can make use of the idle CPU.

Perfect example. I have this 8 CPU beast in the basement, which has 8 700mhz Xeon PIII's in it. For alot of computing tasks, it really stinks because you may as well be running on a single 700 mhz CPU. But, throw a BUNCH of tasks/threads at it and it is a real workhorse.

I think that making sage more SMP friendly would help this problem more than any changes to the DB or engine.

I have no idea how much this is due to the Java underpinnings or if it would take a complete rewrite.

[nielm]

Originally Posted by wvpolekat

The single biggest problem sage has with todays average computer is that it is single threaded.

Sage is definately multi-threaded. Java is natively multi-cpu aware and very multi-threaded. It is actually quite hard to write certain Java tasks in a single-threaded way!

You can see this in windows task manager (enable the 'Threads' column), in my CPU monitor plugin. In process explorer you can see the cpu time per thread. Sage is using 77 threads on my system at the moment (driving a local UI, a placeshifter and an MVP).

Certain intensive tasks which are by nature sequential may only use a single thread...

In general purpose multithreaded applications it is very very hard to use 100% of N Cpu's - there will always be one task that others will be waiting for. Certain technical applications can break processing down into work units and parcel them out to each cpu, but these are normally in specialist domains.

[dvd_maniac]

Only one more day to the huge Intel price cuts and I am thinking of ordering the Q6600 quad core for only $266.00.
This should be a decent upgrade from my current X2 4600+ dual-core. Then maybe a big wiz.bin will be a mute point...