Why large TV recording files are loaded into RAM?

[mkanet]

I was taking a closer look at Windows memory consumption; and, happened to notice Windows 7 file caching mechanism loads gigantic multi-GB media files into cached memory (as they're being recorded) in it's "standby list". I also understand that Windows file caching will aggressively cache most things if free memory is available.

I was wonder if there's a way to prevent these file types (or a directory) from being loaded into cached memory... such as user-defined file type or directory exclusion for the standby list. I searched on google, but didn't see anything that looked related.

I'm also curious if it's only SageTV that does this; or, is this considered normal for other PC based DVR's such as Windows Media Center.

RAMMap can reveal what's specifically loaded into the standby list; and, has a way to purge the list manually by a user. Unfortunately, there's no way to specify exclusions for the standby list.

[gplasky]

Mapped File: Also known as section objects, mapped “views” of files are when the contents of that file are mapped to virtual addresses in memory. This can be a process mapping views of files into its memory (for reading or writing) or for the system file cache.

Also: File Mapping

File mapping is the association of a file's contents with a portion of the virtual address space of a process. The system creates a file mapping object (also known as a section object) to maintain this association. A file view is the portion of virtual address space that a process uses to access the file's contents. File mapping allows the process to use both random input and output (I/O) and sequential I/O. It also allows the process to work efficiently with a large data file, such as a database, without having to map the whole file into memory. Multiple processes can also use memory-mapped files to share data.
Processes read from and write to the file view using pointers, just as they would with dynamically allocated memory. The use of file mapping improves efficiency because the file resides on disk, but the file view resides in memory. Processes can also manipulate the file view with the VirtualProtect function.
This is the normal function of the OS

Gerry

[mkanet]

Thanks Gerry. I now know where the standby list gets it's files from; and, I'm pretty sure I understand their purpose. But it still doesnt make sense why these large media media files are even involved in this process; especially, since there's no real benefit in caching these gigantic mpeg files.

I know that on paper, lower priority standby list items should be purged when needed; with no significant detrimental effect in general system performance. Unfortunately, on my PC, that's not the case. The caching of these media files seem to be the inadvertent reason why my PC has general sluggish performance after multiple recordings over time. I can prove this by manually purging the standby list (when there are multiple media files in this list). The PC is blazing fast after I do that; like I just rebooted the server. I have a commandline utility, EmptyStandbyList.exe, which will do this for me.

I already have 8GB's of RAM; which is plenty for all the processes I have in memory. It would be nice if there was a way to exclude video files from this caching engine; to prevent the inadvertent sluggishness.

In the past, I used to blindly use task manager, not realizing why... even when I closed all my large applications in memory, the system was still sluggish. With utilities like RAMMap and ASP, I know where the problem is and how to fix the symptom at least. I was hoping to figure out a way to prevent this from happening in the first place though; especially, since I can't add anymore RAM to this PC. I know that adding more ram to this PC would also fix the problem as well.

[mkanet]

This article explains every well what I'm seeing; both the symptom and the cause. SageTV opens the media files using a high enough standby list priority that, they don't get purged until they really have to.

According to the article, it's possible to prevent this type of unnecessary file caching by using the FILE_FLAG_NO_BUFFERING flag when open large media files in the source code.

Windows, why your 8 GB of RAM feel like a few MB?

Windows: Disable caching of large files

[Fuzzy]

I love RamMAP, but it can get you in trouble, and chasing things that aren't problems, if you let it. While yes, when sage is recording something, I think the files end up on the priority 5 list. however, as they sit there unaccessed, they get moved down the lower priority lists, and get purged out as necessary. If you are still showing a file that you are not accessing in a low priority list, then that is evidence that no other process has even asked to use that much memory. If a program DOES need that space, then it doesn't take any longer to allocate that space to that request if that is currently used for a low priority standby, than it would if that space was marked free - so it has no real performance impact. I actually would NOT want sage to mark the file access no-cache. That would not only hurt multi-file recording/access performance significantly, but would also make things like live comskip work twice as hard.

If you look under the file details tab, you'll see that those files are probably priority 1, which means that space is effectively free.

Now, It WOULD be nice if sage would access other files in a manner that would force them to stay at high priority standby (like fanart files, for instance), but that's why i use FancyCache on my fanart location, to effectively force those files into RAM after their first access.

If you watch closely (keeping in mind you have to use refresh to update the list), you will see that as a file is recorded, it is on the priority 5 list. As memory is requested though, it is allocated from files held on the lowest list, and then areas of the currently recording file are demoted down to priority 2.

In the end, the thing to think about is that this caching system really does work well - and RAM is cheap.. :-)

[mkanet]

Fuzzy, yes what you explain falls in line with what i read yesterday.

The whole reason I'm interested in purging the standby list is because it does exactly what i hoped it would do.. return my PC back to the fast performance when I first booted up. I while on paper i understand that there should be no performance impact when mapped files try to get removed from the standby list, that's not what happens in reality; at least with a PC thats very busy and limited to only 8GBs of ram. See the links i posted above showing how other people with limited ram (8GBs) have seen the same symptom and used the same solution as me. I just execute a single commandline app; and within a second, my PC runs like it did when it booted up a couple of days before.

Yes, RAM is cheap, but my PC's is maxed out at 8GBs. I'm not ready to buy a new PC yet; especially since the performance issue can easily be addressed with a software based solution..to purge the standby list once a day.

Originally Posted by Fuzzy

I love RamMAP, but it can get you in trouble, and chasing things that aren't problems, if you let it. While yes, when sage is recording something, I think the files end up on the priority 5 list. however, as they sit there unaccessed, they get moved down the lower priority lists, and get purged out as necessary. If you are still showing a file that you are not accessing in a low priority list, then that is evidence that no other process has even asked to use that much memory. If a program DOES need that space, then it doesn't take any longer to allocate that space to that request if that is currently used for a low priority standby, than it would if that space was marked free - so it has no real performance impact. I actually would NOT want sage to mark the file access no-cache. That would not only hurt multi-file recording/access performance significantly, but would also make things like live comskip work twice as hard.

If you look under the file details tab, you'll see that those files are probably priority 1, which means that space is effectively free.

Now, It WOULD be nice if sage would access other files in a manner that would force them to stay at high priority standby (like fanart files, for instance), but that's why i use FancyCache on my fanart location, to effectively force those files into RAM after their first access.

If you watch closely (keeping in mind you have to use refresh to update the list), you will see that as a file is recorded, it is on the priority 5 list. As memory is requested though, it is allocated from files held on the lowest list, and then areas of the currently recording file are demoted down to priority 2.

In the end, the thing to think about is that this caching system really does work well - and RAM is cheap.. :-)

[Fuzzy]

Originally Posted by mkanet

Fuzzy, yes what you explain falls in line with what i read yesterday.

The whole reason I'm interested in purging the standby list is because it does exactly what i hoped it would do.. return my PC back to the fast performance when I first booted up. I while on paper i understand that there should be no performance impact when mapped files try to get removed from the standby list, that's not what happens in reality; at least with a PC thats very busy and limited to only 8GBs of ram. See the links i posted above showing how other people with limited ram (8GBs) have seen the same symptom and used the same solution as me. I just execute a single commandline app; and within a second, my PC runs like it did when it booted up a couple of days before.

Yes, RAM is cheap, but my PC's is maxed out at 8GBs. I'm not ready to buy a new PC yet; especially since the performance issue can easily be addressed with a software based solution..to purge the standby list once a day.

I know others have seen performance improvements - but that is usually on apps where the UI is directly tied to the memory usage (things like photoshop and such). Sage, on the other hand, is very back-buffer insulated from this type of performance hit. What 'performance hit' are you seeing on your sage server that is getting better with the cache clear?

If you really want to force the cache cleared periodically, then you can use a scheduled task to launch your command.

[mkanet]

I know its hard to image; especially, if you're used to seeing the OS operating the way it should; with more than enough RAM. I'll describe as best as I can...

The performance hit is very obvious when using the PC as a workstation. The sluggish/lag effects everything; such as when typing a sentence in notepad.exe (or any other app that requires typing). So, while typing.. there's a .500ms to 1sec lag; sometimes worse. Programs that would open up right away, take longer to load. SageTV clients have more spinning circles in general. Mouse cursor is unaffected.

Before, when I only used task manager (and, all the info it can provide) it used to drive me nuts... I couldnt explain where the lack in system resource was; even when HDD's were mostly idle, all large processes not in memory anymore, and plenty of "available" physical memory. So, I always shrugged my shoulders and just rebooted every 4-10 days to restore the performance back to normal. Even utilities like Process Explorer didn't expose where the problem was.

With RamMap, I can see exactly where the problem is; and, even offers a convenient way to make all the symptoms all across the board go away instantly. Luckily, I have a commandline utility, EmptyStandbyList.exe, that was made specifically for people who have the same exact problem as me. Just execute it; and, silently within a split second, cures the performance related symptoms.

I'm curious if there's a way to prevent the system from putting large files into the standby list in the first place. I've found a couple of utilities that can specify the minimum and maximum NT cache size; but, I dont know if that would do what I'm hoping for. I've also found an MS Windows service that I can installed called, Dynamic Cache; which Im also not sure if it will do what I want.

Originally Posted by Fuzzy

I know others have seen performance improvements - but that is usually on apps where the UI is directly tied to the memory usage (things like photoshop and such). Sage, on the other hand, is very back-buffer insulated from this type of performance hit. What 'performance hit' are you seeing on your sage server that is getting better with the cache clear?

If you really want to force the cache cleared periodically, then you can use a scheduled task to launch your command.

[Fuzzy]

in the XP days, i used a program called CacheMan to tweak the way the windows cache works. However, most the tweaks it used to do are either a part of win7, or were made obsolete in the changes to the cache with win7. It looks like it is still available, and says it's relevant with win7 as well, but I'm not sure what all it does (and of course, it doesn't really state in detail either).

My previous post was asking what slowdowns you are seeing from this, because as a sagetv server, i never experience any slowdowns, and my system routinely runs for a month or so before i end up rebooting it for maintenance/tweaking/not-leaving-well-enough-alone. I do have more RAM than you, but that just means more files remain in RAM at standby priority 1. I often remote into the system via RDP, and use it as a remotely, and don't even notice a lag that way. My system usually only keeps about 50MB available as either Free or Zeroed.

As for preventing the files from being cached - I'm not sure there is much you can do, as sage is directly asking them to be cached by the way it is opening the files.

RamMap is currently stating that since my last reboot (only 12 days ago), it's repurposed about 800GB of priority 1 space - all without my noticing - all with the system being used for Sage, bittorrent, usenet, and running the occasional game via splashtop to my tablet... I have 16GB, zero page file, and things roll along swimmingly...

[wbarber69]

Originally Posted by mkanet

I have a commandline utility, EmptyStandbyList.exe, which will do this for me.

Can you share this utility?

[mkanet]

The creator of ProcessHacker (aka "wj32" on the sysinternals.com forums) made the command line tool called, EmptyStandbyList.exe specifically for people who have this symptom. There's a forum thread here.

Direct Link.
http://www.mediafire.com/?epo2qh9s1ic4q6f

BTW: I just found out that the standby list actually comes from Windows Superfetch service (which also handles prefetching files on HDD); it's not the same as NTFS fileHDD caching like I had thought. So, tools related to limiting cache, setting cache size, etc wouldn't help in this case. I also noticed that my Superfetch service was set at manual, instead of automatic. I just changed the service to Automatic. After doing that , the benefits are apparent (faster application startup and Windows bootup time). I'm "hoping" the standby list performance issue I was seeing was related to this service not running; and, simply enabling this service will solve the weird issue I was having. If not, all I need to do is schedule emptystandbylist.exe to run once a day; which does work very well.

I'll post in a few days when I know for sure if the problem goes away with the service enabled.

EDIT: It looks like enabling Superfetch on Win7 definitely makes my system more responsive; even if my OS and apps are on an SSD. I used to think that Win7 superfetch should be disabled for SSDs; but, that clearly slows things down when disabled. It looks like Superfetch doesnt touch application datasets such as sageTV fanart. If things continue to run just as fast indefinitely, I'll even add a small RAMDisk for my SageTV userdata directory (My relatively large Phoenix central fanart directory is in a separate HDD partition on my SSD).

Originally Posted by wbarber69

Can you share this utility?

[Tiki]

Originally Posted by mkanet

EDIT: It looks like enabling Superfetch on Win7 definitely makes my system more responsive; even if my OS and apps are on an SSD. I used to think that Win7 superfetch should be disabled for SSDs; but, that clearly slows things down when disabled. It looks like Superfetch doesnt touch application datasets such as sageTV fanart. If things continue to run just as fast indefinitely, I'll even add a small RAMDisk for my SageTV userdata directory (My relatively large Phoenix central fanart directory is in a separate HDD partition on my SSD).

Interesting... I know Intel recommends Disabling Superfetch with their SSD's (it's one of the "problems" that the Intel SSD Toolbox checks for). I wonder if enabling Superfetch will shorten the life of your SSD?

[nyplayer]

Originally Posted by Tiki

Interesting... I know Intel recommends Disabling Superfetch with their SSD's (it's one of the "problems" that the Intel SSD Toolbox checks for). I wonder if enabling Superfetch will shorten the life of your SSD?

Also SuperFetch automatically issues a defrag... which is not recommended for a SSD.

[mkanet]

The Disk Defragmenter service can easily be disabled. In fact, I've always had the defrag service disabled; manually running it once in a while on my non-ssd disks.

It looks like most of the real-world performance benefit in Superfetch comes from it's pre-emptive caching to RAM feature; which SSD users can also take advantage of. It certainly made a noticeable difference for my PC's bootup and application startup; especially since my PC has limited memory (8GBs).

Originally Posted by nyplayer

Also SuperFetch automatically issues a defrag... which is not recommended for a SSD.

[mkanet]

BTW: For the record, it looks like Superfetch does not prevent the standby-list performance issue described in the OP. I started to see the symptoms after a couple of days of load tests (scheduling a bunch of simultaneous long recordings).

Scheduling EmptyStandbyList.exeonce a day makes the problem go away. That fix, along with enabling Superfetch keeps my PC running very fast indefinitely; even under heavy load.

[drewg]

Mkanet:

I'm with you. I hate it when the OS tries to be "smart" and wastes tons of RAM caching large files that will be written once, and then probably not read for hours, days, or weeks.

This is one of the reasons I use ZFS on linux -- it allows you to limit, on a per directory basis, the caching policy for the filesystem. So I can choose to have my recordings directory cache only metadata in RAM, rather than the entire file. It also allows you to limit the amount of memory used to cache disk files.

At least on *nix, there is a O_DIRECT option that programs like SageTV can use when opening files in order to prevent the OS from caching them. Not sure about Windows...


Drew

[Fuzzy]

Originally Posted by drewg

Mkanet:

I'm with you. I hate it when the OS tries to be "smart" and wastes tons of RAM caching large files that will be written once, and then probably not read for hours, days, or weeks.

This is one of the reasons I use ZFS on linux -- it allows you to limit, on a per directory basis, the caching policy for the filesystem. So I can choose to have my recordings directory cache only metadata in RAM, rather than the entire file. It also allows you to limit the amount of memory used to cache disk files.

At least on *nix, there is a O_DIRECT option that programs like SageTV can use when opening files in order to prevent the OS from caching them. Not sure about Windows...


Drew

Yes, there is a call that programs can use to bypass the disk caching (well, I think what it actually does is put map the files at priority zero, so it doesn't stay around.

The thing is, even on linux, the file is still mapped to a memory location - that's how the program accesses it - it just forcibly deletes it from memory as soon as the api call is completed, instead of waiting until that memory is needed for something else.

[Taddeusz]

Originally Posted by drewg

I'm with you. I hate it when the OS tries to be "smart" and wastes tons of RAM caching large files that will be written once, and then probably not read for hours, days, or weeks.

So you would rather it waste the RAM by not using it at all? I would rather the OS use free RAM for something useful like caching as long as it doesn't impact the performance of the system.

I can't comment on the performance problem that mkanet is seeing as I have not seen that happen. Of course I also don't double my SageTV server as a workstation. It is a single use system with no monitor, mouse or keyboard.

[drewg]

Originally Posted by Taddeusz

So you would rather it waste the RAM by not using it at all? I would rather the OS use free RAM for something useful like caching as long as it doesn't impact the performance of the system.

Of course I don't want free RAM just for the sake of it, I'm not an idiot. I'd rather the system be smarter about caching decisions, which options like O_DIRECT and advanced filesystems like ZFS allow for. Additionally, many systems have 4GB or less of memory, so attempting to cache multiple 6GB HD recording is just pointless.

The problem is that the LRU cache replacement algorithms used by most OSes for caching files leads to most of the useful stuff (core OS binaries used by interactive users, users files, interactive applications) getting pushed out when a background job is recording gigs of files overnight. The system just sees that all these gigs are actively flowing through it, and that nothing has recently used the interactive stuff, so that stuff just gets pushed aside.

Originally Posted by Taddeusz

I can't comment on the performance problem that mkanet is seeing as I have not seen that happen. Of course I also don't double my SageTV server as a workstation. It is a single use system with no monitor, mouse or keyboard.

I do (or did) double my server as my workstation. Before switching to ZFS, I'd have the same problems as mkanet. Specifically, I'd come into my office in the morning, after watching/recording maybe 30GB of data the previous evening. Almost every morning, I'd touch the mouse and it would take a few seconds for each application to start responding. Watching vmstat made it clear that nearly everything was swapped out.

The problem was that by this point, I had not typed a character on the keyboard in ~16 hours, and most of the core binaries & cache files that I used (email, web browsing, VMs) had been pushed out due to inactivity, in favor of the (mostly useless) data from ~30GB of recordings watched/recorded the previous evening. Worse, the actual application's anonymous memory had been swapped out in many case. All because SageTV (and MythTV before it) had pushed gigabytes of data through the system without setting O_DIRECT.

By using ZFS for my recordings directory, and limiting the ARC (cache) to metadata only, I put an end to this problem. The "important" stuff is still cached fully, and my system is instantly responsive every morning.

Drew

[Fuzzy]

Originally Posted by drewg

Of course I don't want free RAM just for the sake of it, I'm not an idiot. I'd rather the system be smarter about caching decisions, which options like O_DIRECT and advanced filesystems like ZFS allow for. Additionally, many systems have 4GB or less of memory, so attempting to cache multiple 6GB HD recording is just pointless.

The problem is that the LRU cache replacement algorithms used by most OSes for caching files leads to most of the useful stuff (core OS binaries used by interactive users, users files, interactive applications) getting pushed out when a background job is recording gigs of files overnight. The system just sees that all these gigs are actively flowing through it, and that nothing has recently used the interactive stuff, so that stuff just gets pushed aside.

Then you aren't understanding how windows caching works in vista/7/2k8. The caching is not simply LRU (as it was in previous versions). It is actually a priority system, that ultimately functions as a hybrid between traditional LRU and LFU systems. Programs have full granularity on what priority files are opened with, but without that specific call, they are opened at priority 5 (of 0-7). Core system files are usually loaded at priority 6 or 7, so they basically never get purged. The specific problem here is that sage is not setting a priority to the files when they are opened, so they come in at 5, which is likely higher than they really ought to be.

Also, the entire file is not necessarily cached, the cache priority is done on a per-block, not per-file, basis.