Sage, misbehaving big time

[Deacon Crusher]

Long time user, never had a problem like this before. Can't figure if problem is initiating within sage or within opendct, so going to break rules and cross-post for a bit, putting this here and will post in opendct thread as well.

I've tried shutting down entire network as well as sage server and bringing stuff up slowly.

I've updated Java (in the right way with sage shutdown) no help.

Starting around 7:15 last night sage starting throwing the errors I'll list below and after a while (15-30) minutes the sage service is at 95%+ CPU usage and everything stops working. Clients are disconnected and shows start failing to record.

Restart after updates etc. stuff works for 15-30 minutes and then starts to repeat problem.

FYI, many references to network issues, but when sage isn't running the computer it's running on has zero problems including continuous pings and large file copies.

Here are what look like the interesting log passages from opendct and sage.

These are likely just symptoms but damned if I can figure out the cause.

Actually, I'll start with this from the Windows System Log

{Delayed Write Failed} Windows was unable to save all the data for the file \Sage\MSNBCLivewithDavidGura-48618664-0.ts; the data has been lost. This error may be caused by network connectivity issues. Please try to save this file elsewhere.

SAGEtv_0.txt log section right as problem starts
Sun 7/1 13:14:29.167 [Seeker@38ae5f] MARK 1 currRecord=FA[stationID=35975 Speed time=Sun 7/1 12:44:00.000 dur=2:02:00] enc=Hauppauge HD PVR Capture Device clients=[] ir=false
Sun 7/1 13:14:29.175 [Seeker@38ae5f] Seeker in AUTOMATIC mode nextRecord=A[48677476,48520221,"Last Week Tonight With John Oliver",19566@0701.23:00,31,T] nextTTA=35131195
Sun 7/1 13:14:29.183 [Seeker@38ae5f] newRecord=FA[stationID=35975 Speed time=Sun 7/1 12:44:00.000 dur=2:02:00]
Sun 7/1 13:14:29.191 [Seeker@38ae5f] Record was enabled on what's already being recorded; update status.
Sun 7/1 13:14:29.198 [Seeker@38ae5f] Keeping record just the way it is.
Sun 7/1 13:14:29.207 [Seeker@38ae5f] RootFile=H:\ fstype=NTFS fileLength=1476509696 file=H:\Sage-h\Speed-48525414-0.ts
Sun 7/1 13:14:29.215 [Seeker@38ae5f] RootFile=\\ds413\sage-ds413 fstype=null fileLength=2917430624 file=\\ds413\sage-ds413\PGATourGolf-QuickenLoansNationalFinalRound-48650588-0.ts
Sun 7/1 13:14:29.224 [Seeker@38ae5f] RootFile=\\ds413\Sage-ds413-V2 fstype=null fileLength=403777376 file=\\ds413\Sage-ds413-V2\Sage\MSNBCLivewithYasminVossoughian-48618665-0.ts
Sun 7/1 13:14:29.232 [Seeker@38ae5f] RootFile=\\ds413\Sage-ds413-V2 fstype=null fileLength=459065168 file=\\ds413\Sage-ds413-V2\Sage\MLBBaseball-SanFranciscoGiantsatArizonaDiamondbacks-48519375-0.ts
Sun 7/1 13:14:29.241 [VideoFrame-001d6a4bfb0f@1a32879] VF processing job null nPlayin=true
Sun 7/1 13:14:29.249 [Seeker@38ae5f] Checking video directories for new files
Sun 7/1 13:14:29.258 [VideoFrame-001d6a4bfb0f@1a32879] isRec=true rd=868575 base=11973 eos=false
Sun 7/1 13:14:29.266 [Seeker@38ae5f] Diskspace checking is running
Sun 7/1 13:14:29.275 [VideoFrame-001d6a4bfb0f@1a32879] VF thread is now waiting for 0:59:47.102
Sun 7/1 13:14:29.283 [Seeker@38ae5f] Verifying existence of all TV media files in database fixDurs=false avoidArchive=true
Sun 7/1 13:14:29.467 [Seeker@38ae5f] Enforcing keep at most limits for the files...
Sun 7/1 13:14:29.480 [Seeker@38ae5f] DONE enforcing keep at most limits for the files.
Sun 7/1 13:14:29.488 [Seeker@38ae5f] Seeker clearing unwanted and partial files...
Sun 7/1 13:14:29.509 [Seeker@38ae5f] DONE clearing unwanted and partial files.
Sun 7/1 13:14:29.645 [Seeker@38ae5f] needFreeSize = 2.025 GB currFreeSize=1038.389954048 GB
Sun 7/1 13:14:29.654 [Seeker@38ae5f] needFreeSize = 2.025 GB currFreeSize=1529.387747328 GB
Sun 7/1 13:14:29.674 [Seeker@38ae5f] needFreeSize = 4.05 GB currFreeSize=3111.840792576 GB
Sun 7/1 13:14:29.681 [Seeker@38ae5f] Seeker waiting for 40 mins.
Sun 7/1 13:16:21.488 [MediaServerConnection@19f] Error in MediaServerConnection of :java.io.IOException: An unexpected network error occurred
Sun 7/1 13:16:21.497 [Pusher@abb708] Error with MiniPlayer, closing UI: java.io.IOException: An unexpected network error occurred
Sun 7/1 13:16:21.511 [MediaServerConnection@1cf3d77] Error in MediaServerConnection of :java.io.IOException: An unexpected network error occurred
Sun 7/1 13:16:21.520 [MediaServerConnection@9e9de2] Error in MediaServerConnection of :java.io.IOException: An unexpected network error occurred
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] java.io.IOException: An unexpected network error occurred
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sun.nio.ch.FileDispatcherImpl.pwrite(Unknown Source)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sun.nio.ch.IOUtil.write(Unknown Source)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sun.nio.ch.FileChannelImpl.writeInternal(Unknown Source)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sun.nio.ch.FileChannelImpl.write(Unknown Source)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sage.MediaServer$Connection.transferDataUpload(MediaServer.java:960)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sage.MediaServer$Connection.writeFile(MediaServer.java:907)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sage.MediaServer$Connection.run(MediaServer.java:1198)
Sun 7/1 13:16:21.528 [MediaServerConnection@1cf3d77] at sage.Pooler$PooledThread.run(Pooler.java:254)
Sun 7/1 13:16:21.529 [MCSRReconnect@1afd3fa] Preparing the media player for a reload...
Sun 7/1 13:16:21.537 [Pusher@abb708] pushBuffer call failed; terminating push loop

opendct.log as problem starts to happen
13:16:06.611 [NIOHTTPProducerImpl-256:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN SeekableCircularBufferNIO - The buffer has 7863912 bytes left to be read, has only 409 bytes left for writing and 1316 bytes cannot be added. Deferring bytes to queue buffer.
13:16:20.717 [NIOHTTPProducerImpl-450CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN SeekableCircularBufferNIO - The buffer has 7863100 bytes left to be read, has only 1221 bytes left for writing and 1316 bytes cannot be added. Deferring bytes to queue buffer.
13:16:21.548 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] INFO NIOSageTVMediaServer - Sending 'CLOSE' to SageTV server...
13:16:21.549 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] DEBUG FFmpegTransSageTVConsumerImpl - Error cleaning up broken connection => {}An established connection was aborted by the software in your host machine
13:16:21.549 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] INFO NIOSageTVMediaServer - Connecting to SageTV server on socket /127.0.0.1:7818...
13:16:21.549 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] INFO NIOSageTVMediaServer - Sending 'CLOSE' to SageTV server...
13:16:21.549 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] DEBUG FFmpegTransSageTVConsumerImpl - Error cleaning up broken connection => {}An established connection was aborted by the software in your host machine
13:16:21.549 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] INFO NIOSageTVMediaServer - Connecting to SageTV server on socket /127.0.0.1:7818...
13:16:21.549 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] INFO NIOSageTVMediaServer - Sending 'WRITEOPEN \\ds413\Sage-ds413-V2\Sage\MSNBCLivewithYasminVossoughian-48618665-0.ts 154505568' to SageTV server...
13:16:21.549 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] INFO NIOSageTVMediaServer - Sending 'WRITEOPEN \\ds413\sage-ds413\PGATourGolf-QuickenLoansNationalFinalRound-48650588-0.ts 2038711829' to SageTV server...
13:16:21.561 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] INFO NIOSageTVMediaServer - Sending 'CLOSE' to SageTV server...
13:16:21.561 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] DEBUG FFmpegTransSageTVConsumerImpl - Error cleaning up broken connection => {}An established connection was aborted by the software in your host machine
13:16:21.561 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] INFO NIOSageTVMediaServer - Connecting to SageTV server on socket /127.0.0.1:7818...
13:16:21.562 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] INFO NIOSageTVMediaServer - Sending 'WRITEOPEN \\ds413\Sage-ds413-V2\Sage\MLBBaseball-SanFranciscoGiantsatArizonaDiamondbacks-48519375-0.ts 748271409' to SageTV server...
13:16:22.852 [NIOHTTPProducerImpl-256:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN SeekableCircularBufferNIO - The buffer has 7863912 bytes left to be read, has only 409 bytes left for writing and 1316 bytes cannot be added. The queue buffer is full at 31457664 bytes.
13:16:22.880 [NIOHTTPProducerImpl-241CT-HDHomeRun Prime Tuner 131EDBD6-0] WARN SeekableCircularBufferNIO - The buffer has 7863100 bytes left to be read, has only 1221 bytes left for writing and 1316 bytes cannot be added. Deferring bytes to queue buffer.
13:16:23.486 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN mpegts - Repeated 66 times: [395c83c0] PES packet size mismatch.
13:16:23.486 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN mpegts - [37f3ec80] PES packet size mismatch
13:16:23.492 [NIOHTTPProducerImpl-241CT-HDHomeRun Prime Tuner 131EDBD6-0] INFO SeekableCircularBufferNIO - Recovered 55272 bytes from the queue buffer.
13:16:23.508 [NIOHTTPProducerImpl-450CT-HDHomeRun Prime Tuner 131EDBD6-1] INFO SeekableCircularBufferNIO - Recovered 167132 bytes from the queue buffer.
13:16:23.697 [NIOHTTPProducerImpl-256:HDHomeRun HDHR4-2US Tuner 10437101-0] INFO SeekableCircularBufferNIO - Recovered 656684 bytes from the queue buffer.
13:16:23.697 [NIOHTTPProducerImpl-256:HDHomeRun HDHR4-2US Tuner 10437101-0] INFO SeekableCircularBufferNIO - Lost 1504188 bytes that could not be queued in the queue buffer.
13:16:28.983 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN mpegts - Repeated 4 times: [37f3ec80] PES packet size mismatch.
13:16:28.983 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN mpegts - [395c83c0] PES packet size mismatch
13:16:36.565 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN NIOSageTVMediaServer - Unable to send 'WRITE 3160195888 122840' because the socket has not been initialized.
13:16:36.565 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
13:16:36.565 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] DEBUG FFmpegTransSageTVConsumerImpl - Error cleaning up broken connection => {}The socket is not available.
13:16:36.565 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] INFO NIOSageTVMediaServer - Connecting to SageTV server on socket /127.0.0.1:7818...
13:16:36.566 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] INFO NIOSageTVMediaServer - Sending 'WRITEOPEN \\ds413\sage-ds413\PGATourGolf-QuickenLoansNationalFinalRound-48650588-0.ts 2038711829' to SageTV server...

wrapper.log as problem starts

INFO | jvm 1 | 2018/07/01 13:16:06.696 | 13:16:06.611 [NIOHTTPProducerImpl-256:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN SeekableCircularBufferNIO - The buffer has 7863912 bytes left to be read, has only 409 bytes left for writing and 1316 bytes cannot be added. Deferring bytes to queue buffer.
INFO | jvm 1 | 2018/07/01 13:16:20.741 | 13:16:20.717 [NIOHTTPProducerImpl-450CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN SeekableCircularBufferNIO - The buffer has 7863100 bytes left to be read, has only 1221 bytes left for writing and 1316 bytes cannot be added. Deferring bytes to queue buffer.
INFO | jvm 1 | 2018/07/01 13:16:22.868 | 13:16:22.852 [NIOHTTPProducerImpl-256:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN SeekableCircularBufferNIO - The buffer has 7863912 bytes left to be read, has only 409 bytes left for writing and 1316 bytes cannot be added. The queue buffer is full at 31457664 bytes.
INFO | jvm 1 | 2018/07/01 13:16:22.969 | 13:16:22.880 [NIOHTTPProducerImpl-241CT-HDHomeRun Prime Tuner 131EDBD6-0] WARN SeekableCircularBufferNIO - The buffer has 7863100 bytes left to be read, has only 1221 bytes left for writing and 1316 bytes cannot be added. Deferring bytes to queue buffer.
INFO | jvm 1 | 2018/07/01 13:16:23.575 | 13:16:23.486 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN mpegts - Repeated 66 times: [395c83c0] PES packet size mismatch.
INFO | jvm 1 | 2018/07/01 13:16:23.575 | 13:16:23.486 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN mpegts - [37f3ec80] PES packet size mismatch
INFO | jvm 1 | 2018/07/01 13:16:29.036 | 13:16:28.983 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN mpegts - Repeated 4 times: [37f3ec80] PES packet size mismatch.
INFO | jvm 1 | 2018/07/01 13:16:29.036 | 13:16:28.983 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN mpegts - [395c83c0] PES packet size mismatch
INFO | jvm 1 | 2018/07/01 13:16:36.618 | 13:16:36.565 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN NIOSageTVMediaServer - Unable to send 'WRITE 3160195888 122840' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:36.618 | 13:16:36.565 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:36.719 | 13:16:36.697 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] WARN NIOSageTVMediaServer - Unable to send 'WRITE 463086864 83096' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:36.719 | 13:16:36.697 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:36.921 | 13:16:36.830 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN NIOSageTVMediaServer - Unable to send 'WRITE 527576128 92308' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:36.921 | 13:16:36.830 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:51.775 | 13:16:51.748 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] WARN NIOSageTVMediaServer - Unable to send 'WRITE 470100204 67680' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:51.775 | 13:16:51.748 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:51.876 | 13:16:51.783 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN NIOSageTVMediaServer - Unable to send 'WRITE 3189696284 127352' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:51.876 | 13:16:51.783 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:51.977 | 13:16:51.876 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN NIOSageTVMediaServer - Unable to send 'WRITE 535694156 68996' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:16:51.977 | 13:16:51.876 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:17:06.830 | 13:17:06.749 [FFmpegTransSageTVConsumerImpl-243CT-HDHomeRun Prime Tuner 131EDBD6-0] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:17:06.830 | 13:17:06.784 [FFmpegTransSageTVConsumerImpl-258:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:17:06.931 | 13:17:06.869 [NIOHTTPProducerImpl-256:HDHomeRun HDHR4-2US Tuner 10437101-0] WARN SeekableCircularBufferNIO - The buffer has 7863100 bytes left to be read, has only 1221 bytes left for writing and 1316 bytes cannot be added. Deferring bytes to queue buffer.
INFO | jvm 1 | 2018/07/01 13:17:06.931 | 13:17:06.877 [FFmpegTransSageTVConsumerImpl-452CT-HDHomeRun Prime Tuner 131EDBD6-1] WARN NIOSageTVMediaServer - Unable to send 'CLOSE' because the socket has not been initialized.
INFO | jvm 1 | 2018/07/01 13:17:08.654 | [GC pause (G1 Evacuation Pause) (young) 83M->13M(128M), 0.0022229 secs]

Any thoughts or help will be greatly appreciated.

thanks

[tvmaster2]

likely unrelated to your problem, but early this morning, one of my tuner setups went crazy. It’s normally set to component/SPDIF, but I started getting “no signal” from that tuner. I had to go into settings and change the audio/video setting to “component” only, which makes absolutely no sense.
First time in ten years this became an issue, and I can’t figure out how to make it right.

[NetworkGuy]

Did this happen Friday (6/29)?

There were multiple major cuts to the fiber of Internet backbone companies. It affected almost every cable / Internet provider.

https://slate.com/technology/2018/06...s-updates.html

[jmonier]

I don't know if it's related, but I've seen "No Signal" problems on random sub-channels after doing a channel scan. This is even though these same sub-channels are OK when doing a "Preview" in "Channel Setup". And, some sub-channels on a physical channel are OK, while others are bad.

And, the problem clears up on it's own within 24 hours.

This problem has only become visible recently because of the necessity of doing channel scans due to the channel repacks which are going on. It may well have been there for a long time.

[tvmaster2]

Originally Posted by NetworkGuy

Did this happen Friday (6/29)?

There were multiple major cuts to the fiber of Internet backbone companies. It affected almost every cable / Internet provider.

https://slate.com/technology/2018/06...s-updates.html

My problem, the tuner inputs being changed/updated, appeared Sunday morning, maybe 9am PDT. Very odd. The input choices for my Colossus tuner expanded to include HDMI / SPDIF choices which weren’t there to my recollection. No idea how it happened.

[Deacon Crusher]

My problems started Saturday night around 7 PM pacific.

No changes to the system in at least a couple weeks prior to that.

If I restart everything and start some recordings and start watching recordings, everything is fine for around 15 minutes, then shit starts to go crazy. errors like in logs listed above, CPU utilization of Sage and/or Java spike way up. Clients get disconnected.

Thanks

[zoundz]

You might consider the possibility that the power supply is going south. It might not be totally failing, but one of the rails may be having troubles. If you have another power supply, swapping them out might be an easy way to troubleshoot.

[Tiki]

Originally Posted by Deacon Crusher

My problems started Saturday night around 7 PM pacific.

No changes to the system in at least a couple weeks prior to that.

If I restart everything and start some recordings and start watching recordings, everything is fine for around 15 minutes, then shit starts to go crazy. errors like in logs listed above, CPU utilization of Sage and/or Java spike way up. Clients get disconnected.

Thanks

This sounds like a failing piece of hardware to me. I’ve seen similar symptoms with a failing hard drive. The drive detects problems and tries to dynamically reallocate sectors, meanwhile its buffers fill up and throughput slows to a crawl, eventually hogging down your whole computer as it queues up disk i/o commands.

I would imagine you could have similar problems if packets were being dropped in your network due to some hardware problem.

I would also imagine that if you had a network encoder that was writing to a disk on another machine, a hard drive problem could cause a bottleneck that looks like a network problem to the encoder.

[tvmaster2]

Originally Posted by Tiki

This sounds like a failing piece of hardware to me. I’ve seen similar symptoms with a failing hard drive. The drive detects problems and tries to dynamically reallocate sectors, meanwhile its buffers fill up and throughput slows to a crawl, eventually hogging down your whole computer as it queues up disk i/o commands.

I would imagine you could have similar problems if packets were being dropped in your network due to some hardware problem.

I would also imagine that if you had a network encoder that was writing to a disk on another machine, a hard drive problem could cause a bottleneck that looks like a network problem to the encoder.

in your opinion, what’s the best way to check SageTV server internal-drives for errors, on a Windows 7, 32 bit system..

[Deacon Crusher]

I agree that this seems/feels like hardware fail, but I've done a ton of testing of the server box with Sage service turned off and have zero issues.

Did continuous pings for hours.

Moved 30 GB files around between the drives on the sage box and the drives on my home NAS that are locations where sage writes files to and saw zero issues. Maintained continuous 85 MBps (megabyte, not megabit) per second write speeds to my NAS over the network. So, 30 GB file copied in about 6 minutes, did it multiple times, zero issues.

Harder to stress the RAM (which Sage/Java openDTC) together probably do stress (or at least the 1 gig that gets used by Sage's Java footprint.)

I've also looked and looked in all logs and can't find any indication of drive issues (I have smart reporting turned on for drives on windows box and on the NAS.) I have seen issues in windows logs in the past when a drive is starting to go.

Any thoughts about how to better track down possible culprit?

Thanks

Originally Posted by Tiki

This sounds like a failing piece of hardware to me. I’ve seen similar symptoms with a failing hard drive. The drive detects problems and tries to dynamically reallocate sectors, meanwhile its buffers fill up and throughput slows to a crawl, eventually hogging down your whole computer as it queues up disk i/o commands.

I would imagine you could have similar problems if packets were being dropped in your network due to some hardware problem.

I would also imagine that if you had a network encoder that was writing to a disk on another machine, a hard drive problem could cause a bottleneck that looks like a network problem to the encoder.

[Steve20A]

Originally Posted by tvmaster2

in your opinion, what’s the best way to check SageTV server internal-drives for errors, on a Windows 7, 32 bit system..

You might look at CrystalDiskInfo. I run it on Win 10, but it says it will run on Win 7. Excellent, small program that shows SMART data in an easy to read way. Also drive temps.

I have a data server machine and a media server machine. They are running disks that have been spinning 24/7 since 2011. CrystalDiskInfo saved me from a reassigned sector problem that was killing my data server machine. I had a similar sector issue on my media server machine, but caught it with CrystalDiskInfo before it became a problem.

Oh, yeah. It's free.

[tvmaster2]

Originally Posted by Steve20A

You might look at CrystalDiskInfo. I run it on Win 10, but it says it will run on Win 7. Excellent, small program that shows SMART data in an easy to read way. Also drive temps.

I have a data server machine and a media server machine. They are running disks that have been spinning 24/7 since 2011. CrystalDiskInfo saved me from a reassigned sector problem that was killing my data server machine. I had a similar sector issue on my media server machine, but caught it with CrystalDiskInfo before it became a problem.

Oh, yeah. It's free.

cool, thanks!

[Deacon Crusher]

crystal disk info gave my disks a clean bill of health.

turned off windows firewall and put a bunch more exceptions into Defender to keep it from interfering and zero help. Problem came back within 20 minutes of turning Sage on.

I'm switching to Sonic from comcast this summer as internet service provider (symmetric gigabit fiber for $40 a month). Figured at that point I'd probably drop comcast for TV as well and switch to YouTubeTV (also $40 a month) for my cable channels. Monthly saving around $100.

Was hoping to keep Sage running for the OTA channels, but this hassle is putting that into question as the value proposition/roi goes down hill fast when this kind of stuff happens. Really unlikely to be worth building a whole new machine and spending the time to rebuild my sage setup (just keeping wiz.bin and recordings) given that those OTA channels are also available on YouTubeTV and the shows are mostly available on Hulu.

Hoping it doesn't come to that.

[JustFred]

{Delayed Write Failed} Windows was unable to save all the data for the file \Sage\MSNBCLivewithDavidGura-48618664-0.ts; the data has been lost. This error may be caused by network connectivity issues. Please try to save this file elsewhere.

Based on experience, I'd recommend focusing on the above error msg from the OS. I'd bet that Sage just happens to perturb the underlying issue.

crystal disk info gave my disks a clean bill of health.

What software was used to scan the disk(s) in your NAS?

Clients get disconnected.

That tends to point to a network issue. Ping isn't going to stress the network, so don't expect it to tell you much in this case.

Just for grins, try taking the NAS box out of the equation (Sage-wise). Temporarily use local storage for recordings, and see if that improves the situation.

I've seen network switches "eat" packets under the strangest conditions. Right now I'm nursing along a TrendNet 24-port gigabit switch that randomly goes bazoo.

[Deacon Crusher]

How many things can possibly go wrong at once?

Just thought I'd update. I had all but given up as I spent some hours on the primary possible culprits. Decided to devote one more Saturday to this and since I new this was it, I did the worst version of troubleshooting. I changed everything at once.

Replaced the switch that feeds all my HD network tuners and the sage server.

Deleted all Sage and Sage OpenDCT config/properties files.

Rebooted my entire network, from the cable modem out to all switches, wireless gear etc.

All at once.

Then reconfigured Sage OpenDCT.

Then reconfigured Sage.

Sage is now faster then it's been in years and is working great.

Still however, having extender issues. Thinking a bunch of my extenders are suffering over heating issues.

I have no idea what the real problem was but, ...

Extenders are still being flakey.

Network was having wifi problems, they are gone now, but Sage doesn't use Wifi.

Sage is way, way faster then it's been in years.

So, I think cleaning up SageDCT and Sage Properties files were the main key to getting Sage Server working properly. Network reset seems to have fixed Wifi, so that may have helped with some connectivity problems. Switch replacement may or may not have helped. Ordering some laptop coolers to keep extenders cooler to see if that helps with their stability.

May crack open some extenders and see how guts are looking and if there's any other cooling options I should employ (holes in cases etc.)

Dropping Comcast in the next month or two so Sage will become less important as it will just be for OTA/ATSC broadcast PVR. Will be using YoutubeTV for Sports (Baseball and Basketball games), News (MSNBC etc.), and AMC, FX, ScyFy channel, etc. All via Newest model Rokus which I'm very happy with. Once I've made that switch I'll start working on getting Sage content available via some Roku app so don't have to change input on TV.

Thanks all for ideas/thoughts. As always a great community.