WIZ.BIN 50MB? Tips to clean up years worth of data?

[speck55]

Forgive the obviousness of this post, but forum searches of wiz.bin size or wiz.bin clean up and so on yielded zero results - in fact any search of wiz.bin reformatted wiz.bin to be in double-quotes. Weird.

WIZ seems to be 50MB or so on the server.

I think I have had increasing spinny circles of death, at random - long service start times, and mainly when sitting at a PC client and picking a TV recording to play. Spins for 30 seconds to a minute, then I have to hit play again, then a black window This is years worth of watched/unwatched data, and probably a couple of thousand imported movies whose paths no longer exist, and so on.

I remember a couple of years ago reading a post or two on running metadata tools and/or setting a value in .PROPERTIES to prune stuff out of WIZ, as I move toward moving Sage onto a bit more powerful (ok, way overkill) machine.

Anybody have advice on slimming that DB down?

[Tiki]

I don't think there is much you can do, other than to delete it and start over (which I don't recommend). Wiz.bin is a proprietary / closed database format. Sage exposes some APIs that allow developers to read/write data to/from the database, but I don't think there are any general-purpose repair/cleanup tools.

50MB really isn't much space on a media server (my wiz.bin is currently about 40MB, after running Sage for over 10 years). Consider that a typical 1 hour TV show recorded in HD takes roughly 4-6 GB (more than 1000 times as much space!), and you start to realize that space isn't a concern.

Performance is a different issue. I honestly don't know how much a large wiz.bin file impacts performance. I suspect the scheduling engine creates a bigger hit (if you have a lot of favorites, a lot of channels, and allow Sage to look at the schedule a long way in the future).

Deleting wiz.bin would be one way to see how much impact it has on performance. However, the downside is that wiz.bin stores all your favorites and tracks which shows you've watched (so it doesn't record them again). Losing this data would be a major setback for most people.

[will]

I believe the wiz.bin affects start up time the most. So the larger the wiz.bin the slower the service starts.

I don't know if I am making this up but I recall reading that a few years ago on the forum (your question comes up every now and then).

[stanger89]

Originally Posted by will

I believe the wiz.bin affects start up time the most. So the larger the wiz.bin the slower the service starts.

And limited_carny_init resolves that:
http://forums.sagetv.com/forums/show...ted_carny_init

Only side effect of that is intelligent recordings aren't schedule for a "while" after Sage starts.

The only time I've had trouble with spinning rings and the wiz.bin is when I was using comskip plugin, and some of my paths had changed and I had files in the wiz.bin that were no longer accessible. The comskip plugin would try to find the associated comskip files and would have to wait to timeout since the path didn't exist.

You can fix that by setting:
seeker/disable_root_path_existence_check=true
http://forums.sagetv.com/forums/show...ble_root_path*

It will let Sage remove files that it can't find, regardless of if it can access the path or not (normally it won't remove files it can't access the path for, to preserve network files that are temporarily missing due to network issues).

Just make sure all your network paths are online before you try that.

[waynedunham]

Wow, I'm surprised at how big your Wiz.bin is. I'm using my original Wiz.bin migrated forward through the years and it is only 44mb.
Maybe it is because I do not save much of anything. Watch it and delete it. I have a handful of ripped DVDs from my collection. Ones I watch over and over (Galaxy Quest is one) and a few personal videos recorded on my video camera, phone, etc.
Then I have about a half a page (some folders with multiple files) worth of archived recordings.

I had started down the road of ripping all my DVDs, but then sat back and thought about how many of them I watched multiple times and only ripped those DVDs rather than "wasting" a lot of HD space.
Of course I don't have kids either, so I don't need to keep thousands of hours of TV shows and movies available at a seconds notice based on a child's whim.

[Tiki]

Originally Posted by waynedunham

Wow, I'm surprised at how big your Wiz.bin is. I'm using my original Wiz.bin migrated forward through the years and it is only 44mb.
Maybe it is because I do not save much of anything. Watch it and delete it.

Sounds like your Wiz.bin is about the same size as ours (mine was ~40MB, and OP's was ~50MB).

Deleting shows after viewing won't really have any effect on wiz.bin size. Wiz.bin doesn't store the actual shows - just information/metadata, and that data gets retained for stuff you watch, even after you delete a show.

[reggie14]

50MB does seem a little big. I've been running off the same wiz.bin file since 2003, and mine is still only 28MB. I've noticed the size does fluctuate a bit, but I don't think I've ever been above 50MB. But then again, I've never had a huge number of imported videos- 1000 at most.

I doubt the slowdowns are caused by the wiz.bin being large. Running out of space on the heap might be more likely. Did you increase your java heap size to 1GB?

I think you can "clean" up the wiz.bin by setting the advanced_video_file_recovery setting in sage.properties, but that's really more intended to deal with situations like when your moving Sage to a new computer.

[nyplayer]

I am not sure how Sagetv deals with the WIZ.BIN most databases have to be compressed (Compact) to recover empty space. I know on all my Sqlite DB I do a vacuum for example my SJQ DB. I do this with my NEXTPVR DB also... My Sage wiz.bin is over 50meg. It is not a case of how long you have had the Sage DB but how much you have recorded and number of favorites. Also deletions rename etc... Just because some one has had it longer really has nothing to do with the Size.

So if it is not being maintained within Sage there is really nothing you can do but rebuild it and just import the recordings. Of course you will lose your watched status etc..

if you are feeling brave.

http://forums.sagetv.com/forums/show...05&postcount=3

[waynedunham]

Originally Posted by Tiki

Sounds like your Wiz.bin is about the same size as ours (mine was ~40MB, and OP's was ~50MB).

Deleting shows after viewing won't really have any effect on wiz.bin size. Wiz.bin doesn't store the actual shows - just information/metadata, and that data gets retained for stuff you watch, even after you delete a show.

Then mine should be HUGE! I watch a LOT of recorded TV. It is why I NEED 5-8 tuners. Server A has an HDHR-Prime (3 tuners) 2 HD-PVR/STBs and a single original HDHR for OTA (2 rarely used tuners since I have such horrendous reception here). Server B (my old setup) has an HDHR Prime (3 tuners) and an original HDHR (again, OTA only used as a last resort)
With that, it takes a lot of manual manipulation of setting manual recordings of some favorites to make room for other favorites that don't re-air. I do a lot of that to try and keep the HD-PVR/STB and original HDHR recordings to a minimum. The OTA stuff is definitely hit and miss. One day I'll have 100% full rock solid reception from my attic antenna and another day it will barely tune that same channel. And the HD-PVRs are still kind of flakey, the more you use them the more likely they'll lock up and need a power cycle.
But on many nights all 3 Prime tuners are chock full all night long.
You can tell the summer is coming though. I've only got 50 shows in the queue over the next 72 hours on the "A" server. (and yes, I watch it ALL!)

[Mark57]

Originally Posted by waynedunham

Then mine should be HUGE! I watch a LOT of recorded TV. It is why I NEED 5-8 tuners. Server A has an HDHR-Prime (3 tuners) 2 HD-PVR/STBs and a single original HDHR for OTA (2 rarely used tuners since I have such horrendous reception here). Server B (my old setup) has an HDHR Prime (3 tuners) and an original HDHR (again, OTA only used as a last resort)
With that, it takes a lot of manual manipulation of setting manual recordings of some favorites to make room for other favorites that don't re-air. I do a lot of that to try and keep the HD-PVR/STB and original HDHR recordings to a minimum. The OTA stuff is definitely hit and miss. One day I'll have 100% full rock solid reception from my attic antenna and another day it will barely tune that same channel. And the HD-PVRs are still kind of flakey, the more you use them the more likely they'll lock up and need a power cycle.
But on many nights all 3 Prime tuners are chock full all night long.
You can tell the summer is coming though. I've only got 50 shows in the queue over the next 72 hours on the "A" server. (and yes, I watch it ALL!)

Thread sidebar -- regarding your attic antenna reception. My attic roofing panels have the silver mylar coating on it. I took a drill mounted rotary wire brush and scraped it off where it was in a direct line of site with the antenna farm and my attic antenna, I only needed to do a 3' x 3' area due to the proximity of my attic antenna and the inside of the roof. I used a Digiair Pro meter to test before and after and my signal did improve. All my OTA signals are great and account for 85% of my recordings.

[waynedunham]

Originally Posted by Mark57

Thread sidebar -- regarding your attic antenna reception. My attic roofing panels have the silver mylar coating on it. I took a drill mounted rotary wire brush and scraped it off where it was in a direct line of site with the antenna farm and my attic antenna, I only needed to do a 3' x 3' area due to the proximity of my attic antenna and the inside of the roof. I used a Digiair Pro meter to test before and after and my signal did improve. All my OTA signals are great and account for 85% of my recordings.

I only wish I had your luck. I'm on the "wrong" side of the hill. On the other side of the hill is where all the broadcast antennas are. My house is old and doesn't have any insulation, or covering on the underside of the roof. The only insulation is between the ceiling joists. Nothing in the rafters at all.
The attic antenna is better than an indoor antenna, but not by a whole lot.

I would like to look into an external mast antenna, but hesitate to spend the $$$ if it isn't going to net me a major increase in reception. To give an indication of just how bad my reception is sitting down in "the bowl" I'm in. Most of the antennas I can get a signal from are on top of a mountain only about 10 miles away!

And I have tried the antenna outside as well and it didn't make much difference at all over what I am getting in the attic. I'm going to need a mast that will get me 30-50' off the ground I"m guessing.

[Monedeath]

Originally Posted by waynedunham

Wow, I'm surprised at how big your Wiz.bin is. I'm using my original Wiz.bin migrated forward through the years and it is only 44mb.

The SageTV install I'm responsible for has had it wiz.bin reset once already in the past year to address the same complaints the OP has, and after the rebuild/reimport of everything it already was in the 40mb range, albeit MUCH more responsive.

Then our local provider(Cable One) started making frequent changes to our physical channel lineups once they went full digital, resulting in our CableCard(HD HomeRun Prime) becoming the only reliable means of tuning into their programming offerings.

So the HDHomeRun Prime gets replaced with a Ceton (6) tuner, while a pair of HD HomeRuns were added to grab the locals OTA. (two Hauppauge tuners (an 1800 and 2250) retired in the process) and we're back in the realm of the spining circle once more. Of course, we've gone from 9 tuners to 8 tuners to 12 tuners at this point which probably puts our SageTV Server in the category of being one of the larger pure recorders out there in terms of tuners. (Something like multirecord for SageTV would allow retirement/re-purposing for a few tuners as it's mainly record padding that spurred the justification for so many tuners)

I suspect part of our current issue is the overhead with the Ceton tuner as that seemed to be where the server took the single largest and most noticeable performance hit. 32-bit Win7 doesn't seem to have enough addressable memory to handle a large SageTV library + 12 tuners. (A Ceton PCIe (6) tuner card, another Hauppauge 2250 card, and its assorted collection of drives.) Moving to a 64 bit install and more RAM is now on that server's agenda, just have to work out the logistics on how/when I'm going to implement that shift.

In the meantime, other recordings and content are being shuffled off to a newer Plex/File Server that exists in space that SageTV is being kept unaware of to try to help bring the wiz.bin down to a more manageable size, with 26,616.28GB worth of recordings currently tracked in Sage, that might take awhile.

[waynedunham]

Originally Posted by Monedeath

The SageTV install I'm responsible for has had it wiz.bin reset once already in the past year to address the same complaints the OP has, and after the rebuild/reimport of everything it already was in the 40mb range, albeit MUCH more responsive.

Then our local provider(Cable One) started making frequent changes to our physical channel lineups once they went full digital, resulting in our CableCard(HD HomeRun Prime) becoming the only reliable means of tuning into their programming offerings.

So the HDHomeRun Prime gets replaced with a Ceton (6) tuner, while a pair of HD HomeRuns were added to grab the locals OTA. (two Hauppauge tuners (an 1800 and 2250) retired in the process) and we're back in the realm of the spining circle once more. Of course, we've gone from 9 tuners to 8 tuners to 12 tuners at this point which probably puts our SageTV Server in the category of being one of the larger pure recorders out there in terms of tuners. (Something like multirecord for SageTV would allow retirement/re-purposing for a few tuners as it's mainly record padding that spurred the justification for so many tuners)

I suspect part of our current issue is the overhead with the Ceton tuner as that seemed to be where the server took the single largest and most noticeable performance hit. 32-bit Win7 doesn't seem to have enough addressable memory to handle a large SageTV library + 12 tuners. (A Ceton PCIe (6) tuner card, another Hauppauge 2250 card, and its assorted collection of drives.) Moving to a 64 bit install and more RAM is now on that server's agenda, just have to work out the logistics on how/when I'm going to implement that shift.

In the meantime, other recordings and content are being shuffled off to a newer Plex/File Server that exists in space that SageTV is being kept unaware of to try to help bring the wiz.bin down to a more manageable size, with 26,616.28GB worth of recordings currently tracked in Sage, that might take awhile.

Be careful how many network folders you have in your SageTV paths (import, recordings, etc). I noticed a lot more spinning circles before I removed some of my network paths looking at other machines (in my houses internal wired gigabit lan).
I'm not sure how that compares with any Wiz.bin size issues, but I know it helped me to limit the Network folders SageTV looks at.

[Monedeath]

Originally Posted by waynedunham

Be careful how many network folders you have in your SageTV paths (import, recordings, etc). I noticed a lot more spinning circles before I removed some of my network paths looking at other machines (in my houses internal wired gigabit lan).
I'm not sure how that compares with any Wiz.bin size issues, but I know it helped me to limit the Network folders SageTV looks at.

Actually not in line with some of our experience. We actually saw some slight improvements in performance once we started breaking things into smaller (sub-directory//media import directory) containers as a precursor to moving it to Plex rather than maintaining the original flat file system Sage is setup to do on its own. Of course, that may have to do with the size of the hard drive being used and the number of files present as to when that quirk can start to crop up.

Performance wise, usually the deciding factor on that is more likely to be the networked computer/devices power saving settings in regards to Hard Drive sleep modes and how long it takes the drive to spin up. I know we have disks that are getting a little long in the tooth and are starting to report 11+ second spinup times on them, and they have been linked to a number of the spinning circle encounters we've had(the media we wanted to watch was on that drive), but as the other health indicators are fine they're simply being watched rather than replaced(that and replacing 3+TB drives isn't exactly cheap in either money or time taken to transition things across).

[KryptoNyte]

I wonder if the wiz.bin file could be cracked open once this goes open source. There is a lot of data in there, and it would be nice to have some semblance of access to it, or control over it. I suppose that exposing wiz.bin to users might also introduce some serious mayhem.

[stanger89]

You have access to it, the SageTV API gives you access/control over all the data in it.

[KryptoNyte]

Then let me ask a more basic question; why can't the metadata be natively embedded in the media file itself as it is with WMC, providing for a slightly more universal way to access that data?

[reggie14]

Originally Posted by KryptoNyte

Then let me ask a more basic question; why can't the metadata be natively embedded in the media file itself as it is with WMC, providing for a slightly more universal way to access that data?

FWIW, I think Sage did start embedding it in the media files with v7. But I think this is mainly used to handle cases where you're transferring files and Sage needs to re-identify a file.

I think there's a lot more data about each recording in the wiz.bin than what can be stored in the embedded metadata, though.

[Monedeath]

Originally Posted by reggie14

FWIW, I think Sage did start embedding it in the media files with v7. But I think this is mainly used to handle cases where you're transferring files and Sage needs to re-identify a file.

I think there's a lot more data about each recording in the wiz.bin than what can be stored in the embedded metadata, though.

There is an I/O consideration also. While it is all well and good to embed the metadata into the content itself. When it comes to larger collections of media, you want that metadata to also exist in its own database, as having your media program poke its head into every single media file that might be relevant to what your looking for in order to poll the metadata contained within every time you initiate a query is horribly inefficient for more than very small media collections.

[KryptoNyte]

1) If there is metadata attached to the file, how do I look at it?

2) Regardless of whether an external database serves the needs of some, it's still beneficial to have the metadata in the video file itself.