A fatal error has been detected by the Java Runtime Environment:

[wayner]

About an hour after I had decided that my unRAID server would now be my production server I seemed to have a major problem.

Sage crashed and produced a hs_err_pid106.log file. Should I post that file here?

In my regular SageTV log file there were a whole bunch of lines that looked like this:

Sun 2/12 15:05:51.086 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126053376
Sun 2/12 15:05:51.106 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126069760
Sun 2/12 15:05:51.117 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126086144
Sun 2/12 15:05:51.117 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126094336
Sun 2/12 15:05:51.138 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126110720
Sun 2/12 15:05:51.149 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126127104
Sun 2/12 15:05:51.150 [Fork-BASE-53879@603488c9] Preloading image in picture viewer's slideshow monitor thread: MediaFile[id=227725 A[227728,227726,"20140116_011204000_iOS",0@0115.20:12,0,P] mask=P host=Hoylake encodedBy= format=TIFF 0:00:00 0 kbps [] /var/media/pictures/Wayne's iPhone 2014/20140116_011204000_iOS.tif, Seg0[Wed 1/15 20:12:03.999-Wed 1/15 20:12:04.000]]
Sun 2/12 15:05:51.151 [HDHomeRun 1040e3e1 Tuner 0-Encoder@e47cbeb] HDHomeRun 1040e3e1 Tuner 0: /var/media/tv/PGATourGolf-ATTPebbleBeachProAmFinalRound-102562-0.mpg 738703765
Sun 2/12 15:05:51.159 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126135296
Sun 2/12 15:05:51.169 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126143488
Sun 2/12 15:05:51.173 [Fork-BASE-53879@603488c9] Loading RawImage of size 802x1024 for MediaFile[id=227725 A[227728,227726,"20140116_011204000_iOS",0@0115.20:12,0,P] mask=P host=Hoylake encodedBy= format=TIFF 0:00:00 0 kbps [] /var/media/pictures/Wayne's iPhone 2014/20140116_011204000_iOS.tif, Seg0[Wed 1/15 20:12:03.999-Wed 1/15 20:12:04.000]]
Sun 2/12 15:05:51.174 [Fork-BASE-53879@603488c9] Loading 802x1024 image from file /var/media/pictures/Wayne's iPhone 2014/20140116_011204000_iOS.tif
Sun 2/12 15:05:51.180 [V4L-Encoder-video2@15d98864] V4LCap /var/media/tv/30for30-CatholicsvsConvicts-675020-1.ts 2126151680

There were about 50,000 entries in my log with that V4L notice in a nine minute span. The playback of this program is screwed up. There is no audio - the video plays for about one second and then pauses - it does this over and over.

Note that the tuner is an HD-PVR. I just attached this tuner to my unRAID server last night and there seem to be at least minor issues with it - such as the first minute of a recording is no good. But this unit worked fine attached to my old SageTV Windows server.

I restarted the SageTV docker and the log file is getting all of these lines again - roughly 10,000 lines per minute.

Anyone have any idea on what is going wrong?

I have one other hs_err_pid file from about a week ago.

Note that while this issue was going on I also have an HDHR recording content.

[EnterNoEscape]

It looks like you might have debug_capture_progress=true in your Sage.properties file for some reason. That's the only way that particular logging happens. You need to change it back to the default of false to make that logging go away.

[wayner]

Yes it was - here are the settings:

db_perf_analysis=false
debug=true
debug_capture_progress=true
debug_cc_sub=false
debug_core_events=false
debug_exif_parser=true
debug_file_format_changes=false
debug_fps_freq=0
debug_logging=false
debug_msgs=true
debug_rem_actions=false
debug_sd=false
debug_subtitles=false

Anything else I should adjust. And would that cause the Java problems crash?

[EnterNoEscape]

Originally Posted by wayner

Yes it was - here are the settings:

db_perf_analysis=false
debug=true
debug_capture_progress=true
debug_cc_sub=false
debug_core_events=false
debug_exif_parser=true
debug_file_format_changes=false
debug_fps_freq=0
debug_logging=false
debug_msgs=true
debug_rem_actions=false
debug_sd=false
debug_subtitles=false

Anything else I should adjust. And would that cause the Java problems crash?

Something is definitely wrong with the setup. The only place that specific logging comes from is within the Java code (I searched the entire project for the constant strings) and the property is only read once, then the resultant variable is never changed again. If you haven't already, you might want to run a memory test.

[wayner]

Is there a memory test plugin with unRAID?

Why do you suspect that it could be memory? Isn't it a case of somehow the wrong setting in the properties file?

[EnterNoEscape]

Originally Posted by wayner

Is there a memory test plugin with unRAID?

Why do you suspect that it could be memory? Isn't it a case of somehow the wrong setting in the properties file?

My reasoning is that if the property default is false, the saved value is false, even if there was a problem interpreting the default value of false, the result would still be false. If a thread somehow accessed this property before it was set, it would be false. The variable that this property is stored in is never changed once it is set which means we are all but certainly not dealing with one thread writing and another thread reading a stale or cached value. The only conclusion I can draw is either the JVM has a bug or you have an issue with bad RAM.

I had my own experience with bad ram a while ago and the manifestation was a very consistent kernel panic, so it's completely possible that bad ram can cause the exact same problem over and over again.

I believe there's a memory test on the flash drive when you boot into it.

[wayner]

As a first step I changed the debug_capture_progress=true back to debug_capture_progress=false.

I did have a kernel panic on Friday so you might be correct. My system has a slightly non-standard RAM config. This was a desktop system that had 8GB (2x4GB) of RAM. When setting this up as an unRAID server a month ago I added 2x8GB of RAM to go to 24GB as I wanted more memory in the server to run VMs, etc.

You are right that MEMTEST is a boot up option on the unRAID OS USB stick. I will try to run that for a few hours tomorrow to see how it goes.

[BobPhoenix]

According to garycase on the unRAID forums you can get memory problems if you use all 4 memory slots on a desktop board. He suggests only using 2. If you talk to him on the unRAID forums he can give you the technical reasons that I cannot.

One of the boot options with unRAID is Memtest86+ instead of the other three unRAID boot options of normal default, GUI mode and Safe mode. You should run the memory test for 24 hours to get a good test. You should have zero errors.

[wayner]

Thanks Bob - I will take a look at that issues with 4 memory slots. I guess I can go down to 16GB. Are you using all 4 slots? I thought that you said that you have a server with massive amounts of memory.

I will try running memtest, I just started getting the rest of the family to use this server so it is a shame that I have to switch the extenders back to the old server(s) again.

[EnterNoEscape]

Originally Posted by BobPhoenix

According to garycase on the unRAID forums you can get memory problems if you use all 4 memory slots on a desktop board. He suggests only using 2. If you talk to him on the unRAID forums he can give you the technical reasons that I cannot.

One of the boot options with unRAID is Memtest86+ instead of the other three unRAID boot options of normal default, GUI mode and Safe mode. You should run the memory test for 24 hours to get a good test. You should have zero errors.

That reminds me of my problems with both DDR and DDR2 when populating all 4 slots. It always caused strange issues. Now that I'm on DDR3, I have all 4 slots populated and no issues.

[Taddeusz]

Originally Posted by EnterNoEscape

That reminds me of my problems with both DDR and DDR2 when populating all 4 slots. It always caused strange issues. Now that I'm on DDR3, I have all 4 slots populated and no issues.

I've never heard of or had that problem. What brand of memory are you buying? Personally, I try to stick with the more well known brands like Kingston, Crucial, or Corsair.

[EnterNoEscape]

Originally Posted by Taddeusz

I've never heard of or had that problem. What brand of memory are you buying? Personally, I try to stick with the more well known brands like Kingston, Crucial, or Corsair.

Those are the kinds of brands I was using (and still have laying around). The issues were rare, but only happened with all four slots populated. The motherboards were all well known good brands too. Looking back I know that some of the issues might have just been that sometimes you can't run RAM at full speed when you have all 4 dimms populated. My first experience was with an Asus nForce motherboard running 4GB of Kingston PC2100 DDR that was listed as compatible. Now that I'm thinking about it, it likely also has to do with density because when I ran 4 512GB DDR sticks, I didn't have any issues either.

[Taddeusz]

Originally Posted by EnterNoEscape

Those are the kinds of brands I was using (and still have laying around). The issues were rare, but only happened with all four slots populated. The motherboards were all well known good brands too. Looking back I know that some of the issues might have just been that sometimes you can't run RAM at full speed when you have all 4 dimms populated. My first experience was with an Asus nForce motherboard running 4GB of Kingston PC2100 DDR that was listed as compatible. Now that I'm thinking about it, it likely also has to do with density because when I ran 4 512GB DDR sticks, I didn't have any issues either.

I have 3 desktop Core 2 systems all running a full 4 DDR2 sticks at their full 800Mhz. I also have a Core 2 era server/workstation system full with 4 DDR2 fully buffered. The desktop systems are all Intel brand motherboards while the server is Supermicro. Not had any issues at all related to memory.

[BobPhoenix]

This is just what I remember so don't quote me on it.

There are a certain number of memory loads available and on desktop boards the number of loads is less then those required with all 4 slots loaded. Most of the time it won't cause a problem but if you get a lot of weird errors that is the first thing I would try. Also you can minimize the problem by using ECC memory which is usually not possible on a desktop board. But the ECC in the memory will correct errors encountered up to a point. The best is buffered ECC memory which reduces the memory loads on the MB below the limit because of the buffer.


Like I said that is purely from memory. Personally I've been using ECC memory and server class MBs for a long time now and yes I have 32GB in all 4 memory slots on one unRAID server, 16GB each in 4 memory slots on 2 other unRAID servers and 128GB in 16 slots on my dual E5 server. All of them are maxed out memory slot wise.

[wayner]

Here is what garycase said on the unRAID forums about this issue:

I'd drop to 2 8GB modules. Memtest won't necessarily find any errors with 4 modules unless you run a LONG time, as the random bit errors you'll get from excessive bus loading are NOT because the memory is bad, but because the bus signaling is degraded and will occasionally result in bit errors as a result.

[wayner]

Originally Posted by BobPhoenix

Like I said that is purely from memory. Personally I've been using ECC memory and server class MBs for a long time now and yes I have 32GB in all 4 memory slots on one unRAID server, 16GB each in 4 memory slots on 2 other unRAID servers and 128GB in 16 slots on my dual E5 server. All of them are maxed out memory slot wise.

What's the benefit of so much memory? Do you run a boatload of VMs?

I am still unclear of the benefits of more memory on unRAID - sure I need a bunch for my JVM in Sage and a little bit more for other dockers. But what will I lose by dropping from 24GB of memory to 16GB? Currently I also have an Ubuntu VM using 1 GB and I am hoping to run WHS2011 in a VM as well and it will want at least 2GB.

[EnterNoEscape]

Originally Posted by Taddeusz

I have 3 desktop Core 2 systems all running a full 4 DDR2 sticks at their full 800Mhz. I also have a Core 2 era server/workstation system full with 4 DDR2 fully buffered. The desktop systems are all Intel brand motherboards while the server is Supermicro. Not had any issues at all related to memory.

My best recollection is that I was using DDR and DDR2 on AMD boards. I don't remember having these problems with any of my Core 2 Duo boards or DDR3 AMD boards. I've gone through tons of hardware over the years, so I can't remember every single case, but there were enough that I clearly remember this issue.

[BobPhoenix]

Originally Posted by wayner

What's the benefit of so much memory? Do you run a boatload of VMs?

I am still unclear of the benefits of more memory on unRAID - sure I need a bunch for my JVM in Sage and a little bit more for other dockers. But what will I lose by dropping from 24GB of memory to 16GB? Currently I also have an Ubuntu VM using 1 GB and I am hoping to run WHS2011 in a VM as well and it will want at least 2GB.

I have 11 VMs defined on the 128GB box. I used to have ESXi loaded on each box before I got the 128GB one. Each of them had one unRAID VM and one Windows VM running SageTV. Now the three originals are unRAID only with one of them using the SageTV docker. The 128GB has a SageTV Docker as well as a Windows SageTV VM.

Linux/unRAID uses memory as cache space. If you have 2 VMs using 3GB of your 16GB I would say you won't loose anything. I like to keep at least 4GB for unRAID use beyond what is needed for the VMs. I usually try to leave about 1 GB of memory per VM for overhead beyond what I allocated to the VM. So for 1GB VM I would leave 2GB and for a 2GB VM I would leave 3GB. If those 2 VMs were your only VMs that would only come to 5GB of memory usage and 11GB for unRAID which is plenty.

When I used ESXi I could only use about 13GB of the 16GB due to overhead. It was just easier to see because before any VMs were assigned it showed only that amount of memory available. This was because I used PCI pass through. Without that on ESXi it would allow all 16GB of use.

My 128GB server came from this thread: https://lime-technology.com/forum/in...77.0;topicseen

[wayner]

I haven't bothered removing the memory yet and the machine has been running fine for the last week. I will likely let it keep going for now and see if the problem reoccurs.