Why large TV recording files are loaded into RAM?

[drewg]

Originally Posted by Fuzzy

Then you aren't understanding how windows caching works in vista/7/2k8. The caching is not simply LRU (as it was in previous versions). It is actually a priority system, that ultimately functions as a hybrid between traditional LRU and LFU systems. Programs have full granularity on what priority files are opened with, but without that specific call, they are opened at priority 5 (of 0-7). Core system files are usually loaded at priority 6 or 7, so they basically never get purged. The specific problem here is that sage is not setting a priority to the files when they are opened, so they come in at 5, which is likely higher than they really ought to be.

Also, the entire file is not necessarily cached, the cache priority is done on a per-block, not per-file, basis.

You're right, I'm a *nix guy, and know little about windows. I said LRU for simplicity.

What priority are things like user application binaries (ie, outlook, firefox, thunderbird, chrome)? What priority are those program's data (mailbox files, web caches, fonts, etc)? I'm guessing it is 5 because with the absurdly complex windows APIs, most people are going to take the default. So it basically boils down to the same problem as on *nix: Without sage (or whatever is accessing large recording files, including comskip, plex, etc), there is a large potential for the large files to screw up interactive work.

At least in *nix, the "cache" is generally a "page cache", also known as a "Unified Buffer Cache", that is shared between anonymous ram, and file backed pages. This concept is probably similar on windows. At least some unixes allow you to set policies on this cache, so that files cannot cause applications to be swapped out. Not sure how windows behaves here either.

Drew

[Fuzzy]

I think the problems people experience have more to do with the page-file usage, than the cache. They are just a little related when system memory is low. With RAM prices over the last 2 years, I've made it a point to install enough RAM to make the pagefile unnecessary, and actively disable it on all my systems. (My sage server, which does a lot more than just sage, has 16GB, my sage clients all have 6-8GB, and my general purpose desktop has 12GB).

The thing is, having both a disk cache, AND a page file, are counterproductive to each other. Storing files in RAM, and swapping data to disk, are mutually wasteful.

[mkanet]

I am able to at least follow what's going on my PC that causes the sluggishness...

I use the utility called ATM to view different priority standby lists. Yes, SageTV recordings first show up in standby list P5 (shown in white color in attached screenshot). For load testing, I started 11 simultaneous TV recordings in sageTV, then took the screenshot.

P5 standby list data starts to move down to P1 block-by-block; depending what I'm doing in other active applications (ie, up new web pages in my web browser or navigate to new sageTV menus with more fanart). See the second attached screenshot.

However... Instead of data reaching P0 standby list and, subsequently getting purged from memory all together, data moves back UP to priority 5 standby list. This process can go back and forth indefinitely (between standby list P5 and P1) until I start new recordings. It is only then that the older recordings get purged from standby lists (replaced by newer recordings).

During this seemingly never-ending cycle, my system progressively gets sluggish in the course of days (depending on how busy my PC is).

So, why is there sluggishness related to this type of activity? Things seem to be moving along the way they should. Whatever it is, it is related to relatively large chunks of data being juggled between standby list P5 and P1; as witnessed by other people who reported this symptom under similar circumstances.

Also, why does the sluggishness go away instantly after I empty the standby list? These two questions are what I can't answer. There are no Microsoft white papers that can explain this symptom.

Originally Posted by Fuzzy

Then you aren't understanding how windows caching works in vista/7/2k8. The caching is not simply LRU (as it was in previous versions). It is actually a priority system, that ultimately functions as a hybrid between traditional LRU and LFU systems. Programs have full granularity on what priority files are opened with, but without that specific call, they are opened at priority 5 (of 0-7). Core system files are usually loaded at priority 6 or 7, so they basically never get purged. The specific problem here is that sage is not setting a priority to the files when they are opened, so they come in at 5, which is likely higher than they really ought to be.

Also, the entire file is not necessarily cached, the cache priority is done on a per-block, not per-file, basis.

[mkanet]

Nope... my pagefile doesnt get used much. If it does, not much of it gets used. Plus, that wouldn't explain why clearing the standby list immediately cures the slugish symptom.

Originally Posted by Fuzzy

I think the problems people experience have more to do with the page-file usage, than the cache. They are just a little related when system memory is low. With RAM prices over the last 2 years, I've made it a point to install enough RAM to make the pagefile unnecessary, and actively disable it on all my systems. (My sage server, which does a lot more than just sage, has 16GB, my sage clients all have 6-8GB, and my general purpose desktop has 12GB).

The thing is, having both a disk cache, AND a page file, are counterproductive to each other. Storing files in RAM, and swapping data to disk, are mutually wasteful.

[Fuzzy]

Originally Posted by mkanet

I am able to at least follow what's going on my PC that causes the sluggishness...

I use the utility called ATM to view different priority standby lists. Yes, SageTV recordings first show up in standby list P5 (shown in white color in attached screenshot). For load testing, I started 11 simultaneous TV recordings in sageTV, then took the screenshot.

P5 standby list data starts to move down to P1 block-by-block; depending what I'm doing in other active applications (ie, up new web pages in my web browser or navigate to new sageTV menus with more fanart). See the second attached screenshot.

However... Instead of data reaching P0 standby list and, subsequently getting purged from memory all together, data moves back UP to priority 5 standby list. This process can go back and forth indefinitely (between standby list P5 and P1) until I start new recordings. It is only then that the older recordings get purged from standby lists (replaced by newer recordings).

During this seemingly never-ending cycle, my system progressively gets sluggish in the course of days (depending on how busy my PC is).

So, why is there sluggishness related to this type of activity? Things seem to be moving along the way they should. Whatever it is, it is related to relatively large chunks of data being juggled between standby list P5 and P1; as witnessed by other people who reported this symptom under similar circumstances.

Also, why does the sluggishness go away instantly after I empty the standby list? These two questions are what I can't answer. There are no Microsoft white papers that can explain this symptom.

This is odd that they are getting brought back up in the priority list. I don't see that in my server - though honestly, most of my recordings are not done by the sagetv process, but by the R5000HD app (which appears to open the files at only P3). It is almost as if the sage process is not actually closing access to the file, and hits it occasionally for some reason.

EDIT: Actually, it seems that it is likely the SuperFetch (the feature, not the service) that is bringing that data back into the priority 5 list as other processes are releasing memory. An unfortunate aspect of superfetch trying to allow 'new' programs to open and have a lot of memory to use, but then when they are done with that memory, it allows it to recache what was repurposed - whether really needed or not. Again, this is a problem with sage not properly opening the files at a low priority as it should. A lower priority file is less likely to be re-cached. You CAN disable SuperFetch with a registry key, without disabling the other features (ReadyBoost, ReadyBoot, ReadyDrive). The key for disabling SuperFetch, is HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters\EnableSuperfetch="0" if you feel the need to try it.

Not related to this problem, I have always wished sage was more 'modular' process wise. Especially on systems of your size. I really prefer the network encoder based system that the R5000 uses, and wish that is how all sage recording was done. A separate process for each 'tuner' that handles it's own tuner, and simply tunes and writes. Then also a separate process for each extender UI. Sage did make a standalone recording process, but it wasn't updated with the newer tuner codes, so it likely won't work with newer stuff (like the HDHR).

[mkanet]

That was a nice thought. But, it looks like MS Superfetching (not the service) can't be disabled. There's still the same standbylist activity/behavior (and respective sluggishness effect)after I made the registry setting change and rebooted. By manually clearing the standbylist, the problem goes away instantly. BTW, 6 of my SageTV recording sources are network encoders... 4 of them R5000, 2 of them DVB-S2.

Originally Posted by Fuzzy

This is odd that they are getting brought back up in the priority list. I don't see that in my server - though honestly, most of my recordings are not done by the sagetv process, but by the R5000HD app (which appears to open the files at only P3). It is almost as if the sage process is not actually closing access to the file, and hits it occasionally for some reason.

EDIT: Actually, it seems that it is likely the SuperFetch (the feature, not the service) that is bringing that data back into the priority 5 list as other processes are releasing memory. An unfortunate aspect of superfetch trying to allow 'new' programs to open and have a lot of memory to use, but then when they are done with that memory, it allows it to recache what was repurposed - whether really needed or not. Again, this is a problem with sage not properly opening the files at a low priority as it should. A lower priority file is less likely to be re-cached. You CAN disable SuperFetch with a registry key, without disabling the other features (ReadyBoost, ReadyBoot, ReadyDrive). The key for disabling SuperFetch, is HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters\EnableSuperfetch="0" if you feel the need to try it.

Not related to this problem, I have always wished sage was more 'modular' process wise. Especially on systems of your size. I really prefer the network encoder based system that the R5000 uses, and wish that is how all sage recording was done. A separate process for each 'tuner' that handles it's own tuner, and simply tunes and writes. Then also a separate process for each extender UI. Sage did make a standalone recording process, but it wasn't updated with the newer tuner codes, so it likely won't work with newer stuff (like the HDHR).

[Fuzzy]

The disabling of the superfetch wont stop the caching, just the recaching of files previously purged. I still think to understand the problem, you need to figure out what is actually causing it. I don't contest it had to do with the cache, but it's odd that most don't have this problem.

[mkanet]

There aren't very many people who use their PC in the same way I do; especially those who have 8GB of RAM. I can't figure out what specifically causing it with the tools I have. Plus, even if I did find what specifically caused the symptom, I doubt there would be a setting I can just turn off or change to fix the problem.

I can rule out what it's not (such as pagefile activity, general CPU or HDD activity, or lack of "available" RAM shown in task manager). I can also at least narrow it down to superfetch activity that's the indirect cause of the symptom. When me (and others with this symptom) can instantly cure the sluggishness by emptying the entire standby list, one would think it would be a huge clue. Unfortunately, there's no tool I can think of that would expose the direct cause of the issue. I guess throwing a ton of more RAM at the problem could make the problem go away; but, I wouldn't be able to explain that either.

Originally Posted by Fuzzy

The disabling of the superfetch wont stop the caching, just the recaching of files previously purged. I still think to understand the problem, you need to figure out what is actually causing it. I don't contest it had to do with the cache, but it's odd that most don't have this problem.