java.lang.OutOfMemoryError? Anyone Else?

[Narflex]

If you still have that Appaloosa.jpg file please submit a bug report and send it in. I'm curious if there's some issue with that image that's causing something in the image loading system to allocate way too much memory (we use libjpeg for this, so it *should* be quite solid....but if we can reproduce it we'll investigate it).

[panteragstk]

Originally Posted by Narflex

If you still have that Appaloosa.jpg file please submit a bug report and send it in. I'm curious if there's some issue with that image that's causing something in the image loading system to allocate way too much memory (we use libjpeg for this, so it *should* be quite solid....but if we can reproduce it we'll investigate it).

Man, I wish I hadn't deleted it. As soon as they identified that file I got rid of it. Sorry. If it comes up again I'll get a report to you guys.

Now that it seems we've figured that one out. How bout my "DX9Renderer NATIVE WARNING (non-FAILURE) line 1067 hr=0x8007000e" error? That is the only other thing that has given me issues with sage. I get "DX9 is freeing texture memory to make room size=133563392 rtMemUse=0" before the failure, but I don't see how a card with 512mb of memory can run out when rendering sage and nothing else. I'll attach a log when I get the chance.

[Audacity]

Originally Posted by stuckless

Without getting into the technical specifics... SageTV is written on top of virtual machine (Java), and as such, a fixed amount of memory is allocated to that virtual machine. SageTV ships with more than enough memory to accomodate itself, but it can't be expected to be configured allow for an unlimited number of plugins??

Ironically, for system security, isolation, etc, virtual machines are becomming all the rage. I remember when Java started out, people criticized it because it was vm based. Now, we are seeing all sorts of VMs being used. The concept isn't new, it's just making a comeback

Isolating processes into separate vms (whether they are java or not) will tend to lead to more stable system overall. Today, in most OSes, a rogue application can consume 100% of the memory and then cause the entire OS to fail (or become unusable). VMs sandbox processes so that doesn't happen. So when a rogue plugin in SageTV consumes all the memory of SageTV and it craps out... the OS is unaffected.... that's a good thing.

I think you're confusing a JVM with software like vmware. Apples and oranges.

A Java VM has a lot more in common with a language runtime or an interpreter than it does with VMs from companies like vmware. I'd agree that runtimes are pretty common today, like Adobe Flash or .NET. I'd also agree that vmware-style virtual machines are used more often today than they were in the past.

Oh, and while VMs also implicitly sandbox the code that is run within it, sandboxing processes can easily be done by using operating system security and privilege level features. This is how IE9 and Chrome operate.

My point is that applications written in .NET, Flash, Python, etc typically don't require the user to correctly predict how much memory the application will need. It's unusual for users to be expected to do this. The only other application that I deal with that requires memory tuning knowledge of the user is the Oracle RDBMS, and thats targeted at DBAs (who have training), it's not targeted at consumers.

Both BTV and Windows Media Center (the other two PVR apps that I've used in the past) run on .NET, which runs under it's CLR which is very similar to a JVM - and they don't have these issues, and that is a good thing. Sage could still sandbox their software and get the benefits you describe without requiring users to correctly guess how much memory will be required by the process.

Just sayin'.

[gplasky]

VMWare and the rest actually virtualize the hardware of the machine. The operating system running inside of a VMWare container (or Parallels or Windows' virtualization containers or Bochs or ...) have varying degrees of awareness of running within a virtualized container. Within VMWare, the operating system has no idea that it is running within a virtual container. The operating system is not modified at all, although specialized drivers are usually installed (most importantly video) to prevent performance problems. Some other VM's don't do full hardware virtualization and instead require the OS inside the container the make special calls to the container in place of the regular hardware calls.

The JVM is not a virtual machine in that sense at all. No hardware other than the processor is virtualized. The JVM is essentially a virtualized CPU plus the same sort of runtime that is included with a C++ or any other object oriented language, plus garbage collection and other necessities. Additionally, of course, Java class files (and JAR files, etc) are not machine code, but an intermediate byte code. So the JVM has to compile or interpret class files (whether contained in a JAR file or not) at runtime, and has the ability to load and find new code dynamically at runtime.

The JVM is called a virtual machine because the JVM definition defines an abstract machine. This includes registers, stack, etc, and the byte code that Java source is compiled to is practically machine code for this virtual machine. The JVM then interprets or compiles this byte code into native machine instructions.

The difference is essentially that the JVM is a virtualized processor and the other virtual machines are virtualized machines (including video card, network, and other external devices and hardware registers)

Gerry

[stuckless]

Originally Posted by Audacity

I think you're confusing a JVM with software like vmware. Apples and oranges.

A Java VM has a lot more in common with a language runtime or an interpreter than it does with VMs from companies like vmware. I'd agree that runtimes are pretty common today, like Adobe Flash or .NET. I'd also agree that vmware-style virtual machines are used more often today than they were in the past.

Oh, and while VMs also implicitly sandbox the code that is run within it, sandboxing processes can easily be done by using operating system security and privilege level features. This is how IE9 and Chrome operate.

My point is that applications written in .NET, Flash, Python, etc typically don't require the user to correctly predict how much memory the application will need. It's unusual for users to be expected to do this. The only other application that I deal with that requires memory tuning knowledge of the user is the Oracle RDBMS, and thats targeted at DBAs (who have training), it's not targeted at consumers.

Both BTV and Windows Media Center (the other two PVR apps that I've used in the past) run on .NET, which runs under it's CLR which is very similar to a JVM - and they don't have these issues, and that is a good thing. Sage could still sandbox their software and get the benefits you describe without requiring users to correctly guess how much memory will be required by the process.

Just sayin'.

Ok. Just for your reference, because I get confused about all this JVM stuff... you can check the wikipedia page about virtual machines (specifically the process virtual machines) which was the type of VM that I was referring to (in my confused state). People need to stop seeing the letters 'VM' and assuming that people are referring to things like hypervisors, and vmware, xen, etc... it's a generic term that spans a big field (I think).

I agree that newer technologies do a decent job of sandboxing, specificialy chrome... the jury is still out on ie9, but given't MS record... I'm not betting it will be much of sandbox.. probably more like sifter

I would bet that if Java were being designed today, it would probably be different. Java was born in 1995 and in some respects, it's showing its age. And unfortunately, Java won't be getting better any time soon, especially given that it is now owned by Oracle

All that being said, Java's ability to limit the amount of memory is not a bad thing. I may be wrong (or confused), but I don't think you can control the amount of memory a process uses on windows (or Linux for that matter). You can control things like how much cpu, but memory seems to be a free for all... which is good when you need it, but but bad when you have a rogue app that decides it's the only process that should get memory

[panteragstk]

Speaking of memory. What would make this "DX9Renderer NATIVE WARNING (non-FAILURE) line 1067 hr=0x8007000e" error come up?

[Audacity]

Originally Posted by stuckless

All that being said, Java's ability to limit the amount of memory is not a bad thing. I may be wrong (or confused), but I don't think you can control the amount of memory a process uses on windows (or Linux for that matter). You can control things like how much cpu, but memory seems to be a free for all... which is good when you need it, but but bad when you have a rogue app that decides it's the only process that should get memory

Following this line of logic, Mac OS X is a serious downgrade from the original generation of Mac OS (1984-2001 versions) because it did away with user configured per-application memory limits.

The problem that you describe is interesting, because aside from SageTV, none of the other applications on my computers have user declared per-process memory limits. If the scenarios you described were common it would a serious reliability problem on my PCs that are running operating systems that don't set per-process memory limits (Linux, MacOS X, Windows). The reality is that I go weeks and in some cases months between reboots, which is usually done for patch reasons. Sure, occasionally you have a misbehaving application that slows down your system, but never so much that I'm unable to kill the process. That is why modern operating systems implement security access levels, preemptive multitasking and protected memory: to protect from buggy applications.

Originally Posted by stuckless

I agree that newer technologies do a decent job of sandboxing, specificialy chrome... the jury is still out on ie9, but given't MS record... I'm not betting it will be much of sandbox.. probably more like sifter

Well, this technology was also used in IE8 so it's been battle-tested, and is the same operating system features that are available to all applications. Mac OS X implements similar features (I believe theirs is based on TrustedBSD, which Chrome used for their Mac version). What I'm getting at is the "safety feature" of having to configure memory limits is either not needed or already provided by other (existing) operating system features. Specifically:

- Modern operating systems have built-in protections against buggy applications completely starving other processes from resources
- Sandboxing isn't needed for Sage (doesn't the Sage Service by default run as a privileged account). In fact, it would be difficult to sandbox SageTV since it needs all sorts of access to your system (network, devices, local storage, etc). Sandboxing is for running untrusted code so that it has barely enough permissions to do what it needs to do.

I agree that the term "virtual machine" is incredibly generic, that's why I prefer to use the term "runtime"; for example the Java Runtime Environment.

At this point some readers may be thinking "that Audacity guy just has a hate-on for Java and he's bringing up crappy things about SageTV being based on Java that cannot be changed". Nope, what I'm suggesting is that since setting per-process memory limits is almost never required by consumer-focused software, it isn't in the user's "mental model" of stuff that they need to manage. That means you'll either have to educate everyone (expensive) or fix the problem. For example, if you could design SageTV so that if, due to plugins or whatever reason these memory limits were exceeded, detect this condition and increase the configured memory limit. Maybe this change would require SageTV to restart itself after reconfiguring memory limits, but that is far better than leaving the end user confused with a broken PVR.

Make it intelligent enough to look at how much system memory is available in the machine when making decisions on how much memory to use, and write some code to guard against some of the edge cases that could produce undesirable results. In short, track how much memory the SageTV process uses historically and adjust this setting accordingly. Making this memory limit self-adjusting will save the user from frustration and save SageTV LLC additional support costs.

Running on Java has pros and cons. Sure, on the 3 platforms SageTV runs on there will be "rough edges" when compared to native software due to the whole Java thing. Some of that you can't do anything about and some of it you can work around, and this is one of those rough edges that I imagine could be worked around.

What is most unhelpful is taking the position "that's not a bug, it's a feature, hurray for language runtimes that party like it's 1995!"

[panteragstk]

So...yeah...Dx9 memory problem...someone...anyone...

[stuckless]

Originally Posted by panteragstk

So...yeah...Dx9 memory problem...someone...anyone...

Well, I thought I'd get clever and google for the problem... unfortunately googling for "DX9Renderer 0x8007000e", simply returns 1 result... this thread

[panteragstk]

Originally Posted by stuckless

Well, I thought I'd get clever and google for the problem... unfortunately googling for "DX9Renderer 0x8007000e", simply returns 1 result... this thread

Yep. Tried that too. Thanks though.

[jorton]

http://www.google.ca/m/search?oe=UTF...0x8007000e+dx9

If you google 0x8007000e + dx9 you get a lot of "out of memory" results...

Are you using XP?

J

[panteragstk]

Originally Posted by jorton

http://www.google.ca/m/search?oe=UTF...0x8007000e+dx9

If you google 0x8007000e + dx9 you get a lot of "out of memory" results...

Are you using XP?

J

Nope. W7 64bit. I did google the error like that and found quite a few results. However they were all related to games that had bugs in them that were later fixed. I'm going to install .15 when I get home today and mess with it this weekend. It may be something that has been fixed. I just wanted to see if anyone had seen it before.

I don't think this is a driver issue just because I used two different video cards from two different companies (intel, nvidia) and the issue came up with both. I'm going to see if there is a new whql driver for nvidia (that doesn't break video playback) and see if that helps.

Any other suggestions?

[robogeek]

Originally Posted by panteragstk

Any other suggestions?

Have you run the DirectX Diagnostic (dxdiag.exe)? I seen some google references that seemed to indicate a possible corrupted DirectX installation could cause that error. I would run that diagnostic and post the log results here. You may not see any obvious failures in the diagnostic or the diagnostic log, but there may be components of the video side of DX that are listed with an unknown or unavailable status which could also be a possible indication of a bad DX installation.

[panteragstk]

Originally Posted by robogeek

Have you run the DirectX Diagnostic (dxdiag.exe)? I seen some google references that seemed to indicate a possible corrupted DirectX installation could cause that error. I would run that diagnostic and post the log results here. You may not see any obvious failures in the diagnostic or the diagnostic log, but there may be components of the video side of DX that are listed with an unknown or unavailable status which could also be a possible indication of a bad DX installation.

Dxdiag shows nothing wrong. I have a dx issue though. TMT3 really makes it come to light. Something is wrong. I'm going to install the latest drivers and I've already reinstalled dx9. Not sure what else to try short of reinstalling windows.

[robogeek]

Originally Posted by panteragstk

Dxdiag shows nothing wrong. I have a dx issue though. TMT3 really makes it come to light. Something is wrong. I'm going to install the latest drivers and I've already reinstalled dx9. Not sure what else to try short of reinstalling windows.

While not really odd, the only thing that stood out to me in the dx log is the number of filters and decoders installed. It's possible the filter and decoder merits are messed up and the playback graph is using a nonstandard mix of filters and decoders which is causing unexpected behavior in DirectX. If you have any old, obscure, or unneeded codecs or decoders installed or if you have any codec packs installed, I would try uninstalling those first. If the errors continue and you're fairly certain it's a DirectX issue, a reinstall of the OS will probably be quicker than trying to track down corrupted or mismatched DX files, bad DX registry entries, or bad filter and decoder merits.

[panteragstk]

Originally Posted by robogeek

While not really odd, the only thing that stood out to me in the dx log is the number of filters and decoders installed. It's possible the filter and decoder merits are messed up and the playback graph is using a nonstandard mix of filters and decoders which is causing unexpected behavior in DirectX. If you have any old, obscure, or unneeded codecs or decoders installed or if you have any codec packs installed, I would try uninstalling those first. If the errors continue and you're fairly certain it's a DirectX issue, a reinstall of the OS will probably be quicker than trying to track down corrupted or mismatched DX files, bad DX registry entries, or bad filter and decoder merits.

I installed new drivers after that last post and haven't had the issue again. It seems it was a simple driver issue.