Why does SD Sometimes Take Hours to Update?

[peternm22]

Most of the time, Schedules Direct takes about 10 minutes to update. I've noticed that it can sometimes take hours to update though. It seems to occur once every week or two.

I've attached a log to this post. The EPG update started 1 hour and 45 minutes ago, and it's still going on. I've seen it take up to 3 hours before.

Any ideas? I do see some errors in the log related to getting headshots. But shouldn't those errors be slowing the EPG update down every time, not just every once in a while?

[peternm22]

I just checked again, and after 4 hours it still wasn't done. Going by the log, it looks like it essentially gave up about 45 minutes ago.

I restarted the SageTV service to try and force it to try again, and the log shows a SAGETV_COMMUNICATION_ERROR.

See attached for the newest log.

[peternm22]

After SageTV retrying on it's own multiple times (and 3 instances of SAGETV_COMMUNICATION_ERROR), it eventually connected to SD and the EPG was updated completely in 10 minutes.

So confusing...

[EnterNoEscape]

The actual update itself in the first log finished in 38 minutes. The long part was downloading headshots which is a part of the update process, but not critical. Not sure why this is a periodic problem. I would have expected it to happen all the time.

Your second log shows some connectivity issue. It looks like it authenticates successfully, but SD didn't want to send you your lineups at that time.

[peternm22]

Thanks for taking a look at the logs. Is there anyway to turn off the headshots downloading?

[EnterNoEscape]

Oddly enough, I didn't add that option because it never took an extraordinary amount of time for me and when you get there, the important data is already loaded, so I didn't think anyone would care about some potentially lengthy process that doesn't result in missing immediate guide data changes.

[peternm22]

I'm still trying to figure out this problem. The status in the webserver was saying "Next EPG Update In: 0.00hrs" for almost 24 hours yesterday when I decided to try and force it. Which seems to have worked.

I'm not sure if this is related or not, but about half of the channels in my EPG go to Feb 18th and the other half to Feb 22nd. Normally most of the data seems to "stop" within the same few hours.

To try and reset the data, I stopped Sage and deleted sdmd5prog and sdmd5sched. I also cleared out the 'expanded_until' setting in sage.properties.

According to the log, the main EPG update seems to have finished at 18:21:44.180. There are then various errors related to headshots as usually.

I also found a few instances of "SDEPG Unable to perform bulk lookup: sage.epg.sd.SDException: SAGETV_COMMUNICATION_ERROR" in the log too. What is the "bulk lookup"? Also, 3 of these errors seem to be roughly 30 minutes apart. Does it wait 30 minutes before trying again?

There doesn't seem to be much indication that the EPG update is still doing anything apart from a few lines of this: [ThreadMonitor@19c1075] Top 10 %CPU (last 300s) EPG:1%

I'm guessing that means that the EPG thread is still processing.

Any ideas? Is the SAGETV_COMMUNICATION_ERROR related to the headshot lookups? Could SageTV be requesting too much (the headshots?), and the Schedules Direct server is just stopping it?

It seems like most of my problems are occurring after the EPG data is downloaded, and while the headshots are trying to get downloaded. I saw another thread on the forum about the headshot errors as well: https://forums.sagetv.com/forums/showthread.php?t=65657

[peternm22]

Just checked the log this morning (see attached for the full log, this includes everything from the previously posted log).

It looks like the EPG update fully completed at at "Sun 2/3 1:56:24.911". Leading up to this, there was one of these errors roughly every half-hour:

SDEPG Unable to perform bulk lookup: sage.epg.sd.SDException: SAGETV_COMMUNICATION_ERROR

As I'm looking at the log, it seems like all of these are related to getting "recommendations" from Schedules Direct. I believe these recommendations are suppose to be used in the TV Editorials section in SageTV, but that's always blank on my system when I check it.

Looking back at the earlier logs I posted in the beginning of this thread, it seems all of these are connection problems. Is there anyway of seeing a bit more detailed about the connection errors?

[EnterNoEscape]

The recommendations broke only weeks after I implemented them because Scheduled Direct just isn't providing enough data to create any recommendations for content you don't already have in your favorites. If time permits, I'll put in a way to skip the headshots unless someone else doesn't beat me to it. It's fairly trivial, but my SageTV dev environment needs to be updated.

I didn't see anything of concern on those logs.