EPG NullPointerException - Page 2

RymCo · #21 09-27-2018, 05:55 PM

Hiw did you end up fixing this? My guide is all No Data but I know I have an active SD subscription and I did find an error in the logs somewhere talking about NullPointer.

I thought maybe it was bad schedule and it would remedy itself but a month later and it hasn't.

graywolf · #22 09-30-2018, 07:06 AM

I still have issues with No Data holes on my 1 machine (others are fine).
Unfortunately I haven't had much time to dig into it more.
The issue where wiz.bin would lock hasn't reoccurred (knock on wood).

What I do periodically (pain in the ***) is:
1. Stop Sage
2. delete the sdmd5prog & sdmd5sched files
3. edit sage.properties file removing the following lines:
epg_data_sources/<tuner number>/expanded_until=1538316079220
epg_data_sources/<tuner number>/last_run=1538229679219
epg_data_sources/<tuner number>/server_update_id=
4. Restart Sage and let it do a complete rebuild of the EPG data

This is usually good for a couple days then I start getting No Data holes again

wnjj · #23 09-30-2018, 08:25 AM

Quote:

Originally Posted by graywolf

I still have issues with No Data holes on my 1 machine (others are fine).
Unfortunately I haven't had much time to dig into it more.
The issue where wiz.bin would lock hasn't reoccurred (knock on wood).

What I do periodically (pain in the ***) is:
1. Stop Sage
2. delete the sdmd5prog & sdmd5sched files
3. edit sage.properties file removing the following lines:
epg_data_sources/<tuner number>/expanded_until=1538316079220
epg_data_sources/<tuner number>/last_run=1538229679219
epg_data_sources/<tuner number>/server_update_id=
4. Restart Sage and let it do a complete rebuild of the EPG data

This is usually good for a couple days then I start getting No Data holes again

Are you running the null pointer fixed jar I shared? If that seems to fix at least the wiz.bin issue it’s worth getting put into the main code base.

JREkiwi · #24 09-30-2018, 11:21 PM

Quote:

Originally Posted by wnjj

Are you running the null pointer fixed jar I shared? If that seems to fix at least the wiz.bin issue it’s worth getting put into the main code base.

I've not had a single instance of the NullPointerException since running with the fixed jar and it was happening probably 4-5 times a week. Thanks

John

graywolf · #25 10-04-2018, 01:21 PM

Yes. Been running the fix jar without issue
I’d say it is ready to go into production

wnjj · #26 10-04-2018, 03:34 PM

Quote:

Originally Posted by graywolf

Yes. Been running the fix jar without issue
I’d say it is ready to go into production

That's good news. I'll see about submitting my changes to Github.

graywolf · #27 10-08-2018, 01:11 PM

Wiz.bin.tmp = 10/6/2018 5:47 PM
Wiz.bak = 10/6/2018 5:46 PM

Code:

Sat 10/6 17:46:18.161 [EPG@16b5df0] Wizard compressing new file with version 87...
Sat 10/6 17:46:18.164 [EPG@16b5df0] Wizard writing out table info for Year
Sat 10/6 17:46:18.164 [EPG@16b5df0] Wizard writing out table info for Network
Sat 10/6 17:46:18.164 [EPG@16b5df0] Wizard writing out table info for Title
Sat 10/6 17:46:18.200 [EPG@16b5df0] Wizard writing out table info for Channel
Sat 10/6 17:46:18.202 [EPG@16b5df0] Wizard writing out table info for Bonus
Sat 10/6 17:46:18.206 [EPG@16b5df0] Wizard writing out table info for People
Sat 10/6 17:46:18.446 [EPG@16b5df0] Wizard writing out table info for SubCategory
Sat 10/6 17:46:18.447 [EPG@16b5df0] Wizard writing out table info for Rated
Sat 10/6 17:46:18.447 [EPG@16b5df0] Wizard writing out table info for ParentalRating
Sat 10/6 17:46:18.447 [EPG@16b5df0] Wizard writing out table info for ExtendedRating
Sat 10/6 17:46:18.447 [EPG@16b5df0] Wizard writing out table info for Category
Sat 10/6 17:46:18.447 [EPG@16b5df0] Wizard writing out table info for PrimeTitle
Sat 10/6 17:46:18.447 [EPG@16b5df0] Wizard writing out table info for Show
Sat 10/6 17:46:19.285 [EPG@16b5df0] Wizard writing out table info for Airing
Sat 10/6 17:46:19.720 [EPG@16b5df0] Wizard writing out table info for ManualRecord
Sat 10/6 17:46:19.735 [EPG@16b5df0] Wizard writing out table info for MediaFile
Sat 10/6 17:46:20.550 [EPG@16b5df0] Wizard writing out table info for Watched
Sat 10/6 17:46:20.562 [EPG@16b5df0] Wizard writing out table info for Agent
Sat 10/6 17:46:20.571 [EPG@16b5df0] Wizard writing out table info for Wasted
Sat 10/6 17:46:20.574 [EPG@16b5df0] Wizard writing out table info for Playlist
Sat 10/6 17:46:20.574 [EPG@16b5df0] Wizard writing out table info for TVEditorial
Sat 10/6 17:46:20.574 [EPG@16b5df0] Wizard writing out table info for SeriesInfo
Sat 10/6 17:46:20.811 [EPG@16b5df0] Exception during Maintenance:java.lang.NullPointerException
Sat 10/6 17:46:20.811 [EPG@16b5df0] java.lang.NullPointerException
Sat 10/6 17:46:20.811 [EPG@16b5df0] 	at sage.SeriesInfo.write(SeriesInfo.java:201)
Sat 10/6 17:46:20.811 [EPG@16b5df0] 	at sage.Wizard.saveDBFile(Wizard.java:7938)
Sat 10/6 17:46:20.811 [EPG@16b5df0] 	at sage.Wizard.maintenance(Wizard.java:2369)
Sat 10/6 17:46:20.811 [EPG@16b5df0] 	at sage.EPG.run(EPG.java:645)
Sat 10/6 17:46:20.811 [EPG@16b5df0] 	at java.lang.Thread.run(Unknown Source)
Sat 10/6 17:46:20.811 [EPG@16b5df0] sage.epg.sd.SDRipper@18e0de7 needs an update in 16:15:51
Sat 10/6 17:46:20.811 [EPG@16b5df0] EPG needs an update in 975 minutes
Sat 10/6 17:46:20.811 [EPG@16b5df0] EPG's works is done. Waiting...

wnjj · #28 10-08-2018, 04:45 PM

Well that sucks. Now that I looked over my "fixed" code yet again, I see another bug.

Part-time code work is tough.

While it does not add any NULL people, the array length is still pre-determined when the array was originally defined. It's not sufficient to just filter the bad elements. The array needs to be the size of the "valid" amount of people. I need to loop through looking for valid people before creating the array, then fill it.

You should have at least had an "addSeriesInfo: Attempting to add null person!" warning message in your logs some time before this crash to help identify the source.

I'll see about (yet another) update for you when I get home.

wnjj · #29 10-08-2018, 09:16 PM

Newest code at https://www.dropbox.com/s/g3p5k0n16v...ix_100918?dl=0

Here's the latest code:

Code:

      if (people == null || people.length == 0)
      {
        series.people = Pooler.EMPTY_PERSON_ARRAY;
        series.characters = Pooler.EMPTY_STRING_ARRAY;
      }
      else
      {
        if (people.length != characters.length)
          throw new InternalError("ERROR length of people & character arrays do not match!");

        int sz = 0;
        for (int i = 0; i < people.length; i++)
        {
          if ((people[i] != null) && (characters[i] != null)) sz++;
        }
        series.people = new Person[sz];
        series.characters = new String[sz];
        for (int i = 0, j = 0; i < people.length; i++)
        {
          if ((people[i] != null) && (characters[i] != null))
          {
            series.people[j] = people[i];
            series.characters[j] = new String(characters[i]);
            j++;
          }
          else if (Sage.DBG) System.out.println("addSeriesInfo: Attempting to add null person or character!  Series=" + series.getTitle() + ", person #" + i);
        }
      }

Note: This jar also has a sortbyairingdate fix I was doing for another user. It shouldn't affect anything for this issue.

graywolf · #30 10-09-2018, 07:56 AM

Yes, found these:

Code:

Line 1359: Sat 10/6 17:33:51.674 [Timer-2875@17f4dee] addSeriesInfo: Attempting to add null person!  Series=The Dragon Prince, person #0
	Line 1360: Sat 10/6 17:33:51.674 [Timer-2875@17f4dee] addSeriesInfo: Attempting to add null person!  Series=The Dragon Prince, person #1

Will apply the new fix later today.

Greatly appreciated your looking into and working on the fix

wnjj · #31 10-09-2018, 08:39 AM

Quote:

Originally Posted by graywolf

Yes, found these:

Code:

Line 1359: Sat 10/6 17:33:51.674 [Timer-2875@17f4dee] addSeriesInfo: Attempting to add null person!  Series=The Dragon Prince, person #0
	Line 1360: Sat 10/6 17:33:51.674 [Timer-2875@17f4dee] addSeriesInfo: Attempting to add null person!  Series=The Dragon Prince, person #1

Will apply the new fix later today.

Greatly appreciated your looking into and working on the fix

That’s good info. Do you know where that show got its guide data? From SD?

Thanks for your patience with my hack job code.

wnjj · #32 10-09-2018, 09:43 AM

Good thing you said later today. I was thinking offline and realized there is a corner case that would probably still crash and yours may be that one. If the array of people has a non-zero size, but ALL of the people in it are NULL, the code will try to allocate an array of size 0. Additionally, the original Sage code never checked if the character array itself was NULL.

EDIT: As it turns out, Java allows arrays of size 0 which ironically is exactly what the Pooler.EMPTY_PERSON_ARRAY is so I guess my previous one would have worked. This one still is safer with regard to a NULL character array.

So one more round please.

Code:

      int sz = 0;
      if (people != null && characters != null)
      {
        if (people.length != characters.length)
          throw new InternalError("ERROR length of people & character arrays do not match!");

        for (int i = 0; i < people.length; i++)
        {
          if ((people[i] != null) && (characters[i] != null)) sz++;
        }
      }
      if (people == null || characters == null || sz == 0)
      {
        series.people = Pooler.EMPTY_PERSON_ARRAY;
        series.characters = Pooler.EMPTY_STRING_ARRAY;
      }
      else
      {
        series.people = new Person[sz];
        series.characters = new String[sz];
        for (int i = 0, j = 0; i < people.length; i++)
        {
          if ((people[i] != null) && (characters[i] != null))
          {
            series.people[j] = people[i];
            series.characters[j] = new String(characters[i]);
            j++;
          }
          else if (Sage.DBG) System.out.println("addSeriesInfo: Attempting to add null person or character!  Series=" + series.getTitle() + ", person #" + i);
        }
      }

New code at: https://www.dropbox.com/s/g3p5k0n16v...ix_100918?dl=0

graywolf · #33 10-09-2018, 11:15 AM

The Dragon Prince - Import Video (Netflix show)
So I would imagine the info came from BMT

Applied the latest (Sage.jar_9.1.9_seriesinfo_nullperson_fix_100918)

We'll see how it goes.

wnjj · #34 10-09-2018, 11:23 AM

Quote:

Originally Posted by graywolf

The Dragon Prince - Import Video (Netflix show)
So I would imagine the info came from BMT

Applied the latest (Sage.jar_9.1.9_seriesinfo_nullperson_fix_100918)

We'll see how it goes.

Looks like a there's a bug in BMT then that created these NULL people.

RymCo · #35 10-15-2018, 07:20 PM

I'm still getting EPG errors trying to load guide data:

Code:

Mon 10/15 21:09:58.184 [EPG@910c30] Saving properties file to C:\Program Files (x86)\SageTV\SageTV\Sage.properties
Mon 10/15 21:09:58.247 [EPG@910c30] Done writing out the data to the properties file
Mon 10/15 21:09:58.257 [EPG@910c30] sage.epg.sd.SDRipper@d61dc3 needs an update in 0:00:00
Mon 10/15 21:09:58.268 [EPG@910c30] EPG needs an update in 0 minutes
Mon 10/15 21:09:58.279 [EPG@910c30] EPG attempting to expand Shaw Direct (Classic) - Satellite - Canada (East) (sdepg)
Mon 10/15 21:09:58.289 [EPG@910c30] expand called on Shaw Direct (Classic) - Satellite - Canada (East) (sdepg) at Mon 10/15 21:09:58.289 expandedUntil=Mon 10/15 21:05:58.113 scannedUntil=Sun 8/17 2:12:55.807
Mon 10/15 21:09:58.300 [EPG@910c30] EPG Expanding Shaw Direct (Classic) - Satellite - Canada (East) (sdepg) at Mon 10/15 21:09:58.300
Mon 10/15 21:11:58.332 [EPG@910c30] SDEPG Exception thrown: Read timed out
Mon 10/15 21:11:58.344 [EPG@910c30] java.net.SocketTimeoutException: Read timed out
Mon 10/15 21:11:58.345 [EPG@910c30] 	at java.net.SocketInputStream.socketRead0(Native Method)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at java.net.SocketInputStream.read(Unknown Source)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at java.net.SocketInputStream.read(Unknown Source)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at sun.security.ssl.InputRecord.readFully(Unknown Source)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at sun.security.ssl.InputRecord.read(Unknown Source)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
Mon 10/15 21:11:58.345 [EPG@910c30] 	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at java.net.HttpURLConnection.getResponseCode(Unknown Source)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sage.epg.sd.SDSageSession.get(SDSageSession.java:196)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sage.epg.sd.SDSageSession.get(SDSageSession.java:172)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sage.epg.sd.SDSession.getAuth(SDSession.java:483)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sage.epg.sd.SDSession.getAuthJson(SDSession.java:690)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sage.epg.sd.SDSession.getAccountLineups(SDSession.java:812)
Mon 10/15 21:11:58.346 [EPG@910c30] 	at sage.epg.sd.SDRipper.extractGuide(SDRipper.java:1195)
Mon 10/15 21:11:58.347 [EPG@910c30] 	at sage.EPGDataSource.expand(EPGDataSource.java:709)
Mon 10/15 21:11:58.347 [EPG@910c30] 	at sage.EPG.run(EPG.java:773)
Mon 10/15 21:11:58.347 [EPG@910c30] 	at java.lang.Thread.run(Unknown Source)
Mon 10/15 21:11:58.347 [EPG@910c30] ERROR Updating EPG Source Shaw Direct (Classic) - Satellite - Canada (East) (sdepg)
Mon 10/15 21:11:58.357 [EPG@910c30] ErrorText:expand called on Shaw Direct (Classic) - Satellite - Canada (East) (sdepg) at Mon 10/15 21:09:58.289 expandedUntil=Mon 10/15 21:05:58.113 scannedUntil=Sun 8/17 2:12:55.807
EPG Expanding Shaw Direct (Classic) - Satellite - Canada (East) (sdepg) at Mon 10/15 21:09:58.311

My SageTV server is running Java 7 Update 79. Should I upgrade that? Not sure what's going on... I've updated to SageTV 9.1.10. Cleared out my Wiz.bin (backup up first) too.

wnjj · #36 10-28-2018, 09:05 PM

Quote:

Originally Posted by graywolf

The Dragon Prince - Import Video (Netflix show)
So I would imagine the info came from BMT

Applied the latest (Sage.jar_9.1.9_seriesinfo_nullperson_fix_100918)

We'll see how it goes.

Has it been crash-free with the latest code?

graywolf · #37 10-29-2018, 05:32 PM

Correct. No crashes since.

wnjj · #38 11-05-2018, 03:59 PM

Quote:

Originally Posted by graywolf

Correct. No crashes since.

The fix for this has been pulled into the main codebase if you want to close the issue on Github.

graywolf · #39 11-09-2018, 03:10 PM

Quote:

Originally Posted by wnjj

The fix for this has been pulled into the main codebase if you want to close the issue on Github.

Thanks.
Closed the Issue on Github

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Web Server recordings - NullPointerException	pjwerdna	SageTV v9 Customizations	1	01-15-2017 05:39 AM
NullPointerException on arrow up/down	michaeldjcox	SageTV Linux	1	08-18-2015 01:07 AM
Sage Client has NullPointerException and won't connect	srothwell	SageTV Software	5	04-21-2009 10:10 PM
NullPointerException in SageMC 6.3.9	Julianus	SageMC Custom Interface	3	04-10-2009 05:15 PM
NullPointerException importing dvd profiler info	skiingwiz	SageMC Custom Interface	1	08-27-2008 07:01 PM

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)