What are your Wiz.Bin and JVM used memory sizes.

[BobPhoenix]

I currently have a server that took between 1/2 and 1 hour to boot and allow my HD300s to connect. Current memory usage: JVM Heap Size (Used/Total/Max): 2093MB/2218MB/3818MB which is approximately what it had 7 minutes after boot as I remember but here is three hours earlier then now: MemStats: Used=2146MB Total=2156MB Max=3817MB. I suspect the large memory usage and slow boot is because of the large Wiz.bin.

Backed up Wiz.bins after upgrading and their sizes and ages:
Wiz.bin.SageTV V9.0.8.429 - 12/4/2016 - 173,655KB
Wiz.bin.SageTV V9.0.11.483 - 1/1/2017 - 258,315KB
Wiz.bin.SageTV V9.0.12.504 - 1/27/2017 - 454,636KB
current Wiz.bak - 2/1/2017 - 851,624KB
current Wiz.bin - 2/2/2017 - 858,395KB
This server has 17,544 recordings

So I got curious what my other servers have:

1. Linux V9.0.12.504 - 460MB/896MB/3818MB -25,068KB normal - but only 1966 recordings - 15 favorites.
2. Win V9.0.13.536 - 1012MB/1038MB/1038MB - 321,919KB large - 10866 recordings 454 favorites - will be switching to Linux for this server.
3. Win V7.1.9.256 - 513MB/854MB/1038MB - 74,103KB normal - 8878 recordings 378 favorites
4. *Linux V9.0.13.536 - 2093MB/2218MB/3818MB - 858,395KB large - 17,544 recordings - 130 favorites - this is the one above.

So with lots of recordings = lots of memory usage - but should it be this high? Also V7 server #3 has almost as many recordings as #2 but 1/4 the Wiz.bin size. The only V9 server that appears to have normal sized Wiz.bin is one that has less then 2000 recordings so maybe that isn't normal after all. I am going to blow away my 800MB Wiz.bin this weekend and start over again. I've done that before and it has always reduced the size of the Wiz.bin to 100MB maybe less and reduced the boot time to just a few minutes but it eventually grows to 800MB+ eventually. I worry about server #3 above. It has to stay on Windows and I don't want the memory usage of server #2 since you can see it is currently pushing maximums.

So what are your usages?

[wnjj]

Yeah. Those are huge wiz.bin sizes. Mine is about 50MB if I remember correct. Not sure how many recordings. This is after 8 years of use and 4TB of current recordings.

Part of what makes wiz.bin larger is that it stores the full path (twice actually for photos) to all of the media files so if you have a lot and in particular if your import paths are lengthy there couod be a lot of data there. It also stores all of the camera settings, etc. for photos.

Did you add new media directories or move where it was stored when those large increases happened?

[Narflex]

Wow...those are the largest Wiz.bin sizes I have ever seen. A good way to help break down where the sizes are being used is to look at the log files. It'll print out the size of each individual table in the database when it loads it...then you'll know where this is coming from.

I'm pretty curious myself, so go ahead and post your log here from when it starts up and I'll help you figure it out (the one with 17k recordings).

[BobPhoenix]

Originally Posted by wnjj

Yeah. Those are huge wiz.bin sizes. Mine is about 50MB if I remember correct. Not sure how many recordings. This is after 8 years of use and 4TB of current recordings.

Part of what makes wiz.bin larger is that it stores the full path (twice actually for photos) to all of the media files so if you have a lot and in particular if your import paths are lengthy there couod be a lot of data there. It also stores all of the camera settings, etc. for photos.

Did you add new media directories or move where it was stored when those large increases happened?

Only videos, BluRay's, DVDs, Reencoded videos, Netflix and Amazon prime recordings. Between when it was 454,636KB and 858,395KB nothing added except thru recordings and quite possibly no reboots since it was only 6 days. I regularly get ~1TB estimated in next 360 hours on that server. Currently shows 205 shows 996GB estimated space usage.

Originally Posted by Narflex

Wow...those are the largest Wiz.bin sizes I have ever seen. A good way to help break down where the sizes are being used is to look at the log files. It'll print out the size of each individual table in the database when it loads it...then you'll know where this is coming from.

I'm pretty curious myself, so go ahead and post your log here from when it starts up and I'll help you figure it out (the one with 17k recordings).

I will reboot the server tonight or tomorrow and get them. The log has wrapped them out due to the "Anti Favorites for Dummies plugin" entries - going to see if I turned on logging unintentionally for it.

Question will this property entry increase the number of logs kept?

num_logfiles_to_keep=4

I suspect a plugin maybe even the mentioned plugin but not sure why it would impact the Wiz.bin so probably it is another one. I'm also going to start from scratch and remove the need for some other plugins as I won't be using my BluRay and DVD import libraries to see if that makes a difference.

[BobPhoenix]

It wouldn't boot. I had to use an older version of the wiz bin for it to load. Used the 400MB version to boot with. Hope this will tell you what you need.

Was too big to post even zipped so hope you can get them from this drop box link:
https://www.dropbox.com/s/x9av2gysqn...izbin.zip?dl=0

[BobPhoenix]

Don't have the 1st boot where I waited over an hour for the "Sage waiting for GC to free up some memory usage" to disappear. But attached is the second and final attempt with the 800MB wiz.bin file. I have now started over and have deleted the wiz bin and properties files and let them get recreated. So if you want reboots and such it would have to be with my 300+MB Windows install. Sorry about that but after using the 400MB wiz.bin I only had 5 days in the schedule and it wouldn't update. So since I was going to do this anyway I did it earlier then planned. I did back the files up so I can do a quick switch back if needed but it wouldn't be this weekend.

[BobPhoenix]

After moving the entire server folder to a safe location and starting completely from scratch with an empty appdata folder I have logs from that boot up at the drop box link if that would help: https://www.dropbox.com/s/kdjt4u8omy...ratch.zip?dl=0

Note it created a Wiz.bin of 110MB but after a restart of the docker it is now down to 55MB in size so as of right now I would say a reinstall fixed my problem on the 800MB server.

I can't really start over again on my Windows server that has 300MB+ Wiz.bin as it has recordings made from v6 of Sage. So no metadata in the file to read and with EPG updates allot of the airingIDs on the media files no longer match any more. So they would loose the metadata if I started from scratch. However I should be able to reinstall v7 sage and restore an old backup of the wiz.bin - I think anyway. Then upgrade. However I was planing on switching that one to linux and I would just start it from scratch and record the same shows as the old Windows server. The Windows server would become a view only server with no recording done so no need for EPG. At least that is my plan.

[Narflex]

From the 400MB version, the clear offender is the Agent table:

Sat 2/4 0:53:15.733 [main@3cd1a2f1] Wizard loading main index for Agent bytes=393150292

I have never seen anything like that happen before...that would never happen under normal SageTV usage (i.e. creating Favorites, SageTV's automatic creation of Agents). There's definitely something funky at work here...and since you mentioned the anti-favorite plugin, that'd be my first guess (although i don't know exactly how it works).

If you set the property: dump_db_on_startup=true

Then it'll write out a text file that contains the information (not complete info, but good for debugging) of the entire database so you can see why there's so many agents in the database.

For clarification..Agents map to Favorites as well as being automatically created by SageTV internally as part of Intelligent Recording analysis. Also, how many Favorites does SageTV say you have?

And the 800MB DB has the same issue, but twice as bad:

Sat 2/4 0:43:54.649 [main@3cd1a2f1] Wizard allocating table for Agent of size 7876642
Sat 2/4 0:43:54.671 [main@3cd1a2f1] Wizard loading main index for Agent bytes=771916253

There's almost 8 million Agents in your database...that's just crazy. As a comparison, my Wiz.bin file is 14 years old now and I only have 98,543 Agents in mine and my entire Wiz.bin file is 127MB.

And you were correct about the num_logfiles_to_keep setting controlling how many log files SageTV will keep.

[tmiranda]

Originally Posted by Narflex

I have never seen anything like that happen before...that would never happen under normal SageTV usage (i.e. creating Favorites, SageTV's automatic creation of Agents). There's definitely something funky at work here...and since you mentioned the anti-favorite plugin, that'd be my first guess (although i don't know exactly how it works).

The plugin periodically scans the guide data for any shows that you have added as an anti-favorite. When it find one it marks the airing as Watched and Don't Like.

How many anti-favorites do you have? I have about 10 and my wiz.bin is not that large.

[BobPhoenix]

Originally Posted by Narflex

From the 400MB version, the clear offender is the Agent table:

Sat 2/4 0:53:15.733 [main@3cd1a2f1] Wizard loading main index for Agent bytes=393150292

I have never seen anything like that happen before...that would never happen under normal SageTV usage (i.e. creating Favorites, SageTV's automatic creation of Agents). There's definitely something funky at work here...and since you mentioned the anti-favorite plugin, that'd be my first guess (although i don't know exactly how it works).

If you set the property: dump_db_on_startup=true

Then it'll write out a text file that contains the information (not complete info, but good for debugging) of the entire database so you can see why there's so many agents in the database.

For clarification..Agents map to Favorites as well as being automatically created by SageTV internally as part of Intelligent Recording analysis. Also, how many Favorites does SageTV say you have?

And the 800MB DB has the same issue, but twice as bad:

Sat 2/4 0:43:54.649 [main@3cd1a2f1] Wizard allocating table for Agent of size 7876642
Sat 2/4 0:43:54.671 [main@3cd1a2f1] Wizard loading main index for Agent bytes=771916253

There's almost 8 million Agents in your database...that's just crazy. As a comparison, my Wiz.bin file is 14 years old now and I only have 98,543 Agents in mine and my entire Wiz.bin file is 127MB.

And you were correct about the num_logfiles_to_keep setting controlling how many log files SageTV will keep.

I have started over again on the 800MB server and it is behaving itself currently at only 42MB wiz.bin. I also have kept the plugins to a minimum so far. I'm going to add plugins back slowly as I need them and see what could be creating so many agents. If I had bigger recording window I would switch back from the backup sooner than this coming weekend. I will restore my SageTV server at that time and turn on the dump_db_on_startup=true at that time. I can say what I remember for a favorite count was close to the same as now as I used the Web server to export and then restore after my nuke and pave. It is 132 favorites now and my first post shows it as 130 so I added 2 after the nuke and pave. My misbehaving v9 Windows server has 454 favorites. My working v7 Windows server has 381 favorites I've added 3 since my first post.

Originally Posted by tmiranda

The plugin periodically scans the guide data for any shows that you have added as an anti-favorite. When it find one it marks the airing as Watched and Don't Like.

How many anti-favorites do you have? I have about 10 and my wiz.bin is not that large.

I would bet that I had several hundred but I will check when I restore this weekend to test the dump_db property for Jeff. I have two keyword favorites of "Premiere" and "Pilot" to catch all the new shows but they catch many that I don't want so I add them to my anti favorites list. On my 7.1.9 server that is behaving itself I have 373 anti favorites. On my Windows v9 server that isn't behaving itself I have 139. So Anti-Favorites isn't likely a cause of the large Wiz.bin files unless it only does it on v9 servers.

And just so you both know I have almost identical plugins installed on all of my servers except for the one I started over. On that one I have the following: Commercial Detector, Jetty, SageTV Web server, Comskip, Encoder Names & Restart from Setup. I kept them to a minimum so far.

On the Windows v9 server that is misbehaving I have: Anti-Favorites, Commercial Detector, Complementary Metadata, Comskip, CPU meter, Encoder Names, Disk Usage, Java Heap, Restart from setup, Jetty, SageTV Web server, Phoenix core, SageX.

On the working 7.1.9 server has more because I used to use it for playback as well: Anti-Favorites, BluRay/DVD external, Commercial Detector, Complementary Metadata, Comskip, CPU Meter, Encoder Names, Disk Usage, Java Heap, Restart from setup, Signal Strengh on OSD, BMT, Jetty, SageTV Web server, Phoenix core and SageX.

What it looks like to me is that each time I installed a new version of V9 server it duplicated+ the data in the Wiz.bin for the Agents at least. It may have started in v7 before I converted to the linux server as I don't have any wiz.bin's less than 173MB for v9. I do have a backup from 4/24/2016 with a Wiz.bin when it was on windows of 76MB but I don't have any backups from Windows any sooner or the initial from linux any more.

[JREkiwi]

Just wondering which anti-favorites plugin you're using.
Favorite Excluder (Anti-Favorites)
Anti-Favorites for Dummies

John

[BobPhoenix]

Originally Posted by JREkiwi

Just wondering which anti-favorites plugin you're using.
Favorite Excluder (Anti-Favorites)
Anti-Favorites for Dummies

John

Anti-Favorites for Dummies