SageTV 7.1.9 deleting massive numbers of recordings

[ehfortin]

Hi,

SageTV is driving me crazy for the last few days. It started doing massive delete all over the place. It just destroy nearly a third of my recordings. I'm not missing space at all (have about 5 TB available). Here is what appear in the log:

Mon 11/28 19:27:14.068 [Seeker@1076fe5] MediaFile failing verify testFile=\\fileserver\recorded\sagetv\TheYoungVictoria-5600408-0.ts isFile=false len=0 isTV=true isRecording=false
Mon 11/28 19:27:14.068 [Seeker@1076fe5] Removing MediaFile because it failed verification:MediaFile[id=7662774 A[5600408,5536132,"The Young Victoria",33382@0823.12:50,110] mask=TV host=svcserver encodedBy= format=MPEG2-TS 1:49:50 2884 kbps [#0 Video[H.264 59.94006 fps 1280x720 16:9 progressive id=1011]#1 Audio[AC3 48000 Hz 2 channels 384 kbps MAIN idx=1 id=1100]{Actor=Emily Blunt;Rupert Friend;Paul Bettany;Miranda Richardson;Jim Broadbent;Thomas Kretschmann;Jesper Christensen;Mark Strong;Jeanette Hain;Julian Glover;Michael Maloney;Michiel Huisman;Genevieve O'Reilly;Rachael Stirling;Michaela Brooks, Director=Jean-Marc Vallée}] \\fileserver\recorded\sagetv\TheYoungVictoria-5600408-0.ts, Seg0[Mon 8/23 12:50:09.440-Mon 8/23 14:40:00.000]]
Mon 11/28 19:27:14.104 [Seeker@1076fe5] Deleted media segment file \\fileserver\recorded\sagetv\TheYoungVictoria-5600408-0.ts dtime=35 len=0

Does anybody knows how to stop SageTV from doing this? I'm unable to find any rules that says "Do not destroy anything". I want to delete stuff myself. And no, those are not marked for deletion or "Watched" (some are for sure but most are not).

I suppose some files may have been deleted in the past but not at the current rate which is just alarming. And as I really don't miss space, it just don't make sense that any files is deleted at all.

Any idea/suggestions?

Thanks.

ehfortin

[darcilicious]

Searching the forums on "Removing MediaFile because it failed verification" seems to indicate that it's a hardware failure of some sort -- either the tuner or hard drive possibly.

http://forums.sagetv.com/forums/showthread.php?t=40906 for one example...

[ehfortin]

It can't be related to bad harddrive as it's stored on a NAS that is working fine (no errors, no lost file, no corrupted file). It would be strange as well that the HD-PVR is having anything to do with this as it has been working great for the last few years and the show that are removed can be either old or new. Deletion seems to be regardless of date it was recorded. And HD-PVR is still recording and the show are working.

In order to stop SageTV of doing this, If I put all my recording as a "media folder", will SageTV recognize those as recordings anyway but won't be allowed to delete them?

Right now, it's the second time I'm starting SageTV since saturday and it's the second time it's deleting numerous show. It start right after the service is running. I have to ask this: is it possible that SageTV created a file list to delete based on some criteria and, as I stopped it yesterday when I realized it was deleting lot of recordings and restarted it (without my SageTV recordings folder active). Now, I added the folder back and it started to delete again but stopped by itself after about 50 shows. Is it because it reach the end of the list for now or I may loose another bunch of files if I have to do a stop/start again?

Thank you.

ehfortin

[darcilicious]

Honestly, I have no direct experience with this and was just passing on the thread as something that might be helpful. If you're not seeing hardware as a possible issue then I have no additional hints, sorry.

You might be better off sending your logs along to SageTV support http://www.sagetv.com/request.html

[PosterBoy]

Are you running the Plex media server on the files. I saw sage delete a ton of files for some reason when Plex was doing a library scan.

[CarlR]

I saw the same error and file deletion back when I was running a defrag operation on my recording drives - it seemed that if Sage tried to access a file while it was locked by the defragger Sage would think the file was damaged.

I never found a solution. I just stopped doing defrag on the Sage drives.

[Savage1701]

I had a similar event occur when I made the mistake of running defragmention while the Sage app/service was running. I use a RAID controller and immediately suspected a media error or dropped drive, but in this case it correlated exactly to when I ran a defrag utility.

I do not run defragmentation on any drive on my Sage Server, including the system drive, when Sage is running. For that matter, I don't really run it at all since my Sage Server is essentially a static appliance I reboot every now and then.

[ehfortin]

Ok. I've not done a defrag as it is on a NAS but I can't be sure if it's not something similar with files being locked by something else or being modified. I edited some files at some point to fix an issue with video changing resolution after a few seconds (I'm using an HD-PVR and I'm recording both from SD and HD station so I can have recording that start in SD and then move to HD once the channel is changed). SageTV is handling this correctly but other player really don't like it (XBMC, Media Player and a few others are not liking this at all). I'm pretty sure I've not touched that much file but even if it was the case, I don't understand why it would start doing this at least 3-4 months later. And those files were working fine before and after the removal of the first 4-5 seconds.

As it seems I may not be able to control this specifically for the recordings, is there a way to make sure it won't happens? I'm thinking about converting my remaining files to MKV. I think I read that a converted file would not be deleted, is that accurate?

Or what if I would "rename" the .ts to the actual movie name as XBMC does it (aka "Full Name (YEAR).ts"), would it be recognize as "imported media" and stop the devasting delete rage?

Thank you.

ehfortin

[ehfortin]

Hi,

Originally Posted by PosterBoy

Are you running the Plex media server on the files. I saw sage delete a ton of files for some reason when Plex was doing a library scan.

No. However, with that information and the thing with defrag, I can see a potential cause for this. It is just frustrating that I don't know what is the direct cause so I can't fix it at the source. I either have to find a workaround (like the MKV or renaming of files to import them as new media) or an option in the sagetv.properties that would stop SageTV to delete anything for any reason except if I force a delete.

Thank you for your post, it was a good hint.

ehfortin

[BobPhoenix]

Originally Posted by ehfortin

Ok. I've not done a defrag as it is on a NAS but I can't be sure if it's not something similar with files being locked by something else or being modified. I edited some files at some point to fix an issue with video changing resolution after a few seconds (I'm using an HD-PVR and I'm recording both from SD and HD station so I can have recording that start in SD and then move to HD once the channel is changed). SageTV is handling this correctly but other player really don't like it (XBMC, Media Player and a few others are not liking this at all). I'm pretty sure I've not touched that much file but even if it was the case, I don't understand why it would start doing this at least 3-4 months later. And those files were working fine before and after the removal of the first 4-5 seconds.

As it seems I may not be able to control this specifically for the recordings, is there a way to make sure it won't happens? I'm thinking about converting my remaining files to MKV. I think I read that a converted file would not be deleted, is that accurate?

Or what if I would "rename" the .ts to the actual movie name as XBMC does it (aka "Full Name (YEAR).ts"), would it be recognize as "imported media" and stop the devasting delete rage?

Thank you.

ehfortin

This sounds like you are putting the edited files back in a recording directory on you NAS. If so I would create directories on your NAS that are ONLY imported directories and put the edit files there and delete the originals from the recording directory. If this is not the case then ignore this post.

[ehfortin]

Originally Posted by BobPhoenix

This sounds like you are putting the edited files back in a recording directory on you NAS. If so I would create directories on your NAS that are ONLY imported directories and put the edit files there and delete the originals from the recording directory. If this is not the case then ignore this post.

That was the case. Can you confirm that if I put all my recording in a directory that is imported, SageTV will link them to former recording based on the name ("title-id.ts")? And if I have some .edl file, will they still work once I "move" everything in an imported folder?

Thank you.

ehfortin

[BobPhoenix]

Originally Posted by ehfortin

That was the case. Can you confirm that if I put all my recording in a directory that is imported, SageTV will link them to former recording based on the name ("title-id.ts")? And if I have some .edl file, will they still work once I "move" everything in an imported folder?

Thank you.

ehfortin

Yes it should - can't say will. I get a few that don't reconnect (maybe 1% at most). In earlier versions of SageTV before the metadata was written to the files they would then become imported videos. Now with V7 they are still listed in recordings. They have lost the recorded channel info (that 1%). The description is still present (all 100%). EDL files will still work in imported directories.

Opps forgot to post how I do this.

1. Put edited file on same drive as the imported folder but not in folder that is below the imported folder
2. Stop SageTV
3. Delete original recording
4. Move recording from temporary location to import folder - this should be quick
5. Restart SageTV and let it finish scanning your import folders
6. Once done go to Archived Videos and find your files. If you don't want them in Archived Videos then just toggle the archive flag

Note I have 20 - 30 TB of videos on my unRAID server that are still connected by following the above steps.

[ehfortin]

Originally Posted by BobPhoenix

Yes it should - can't say will. I get a few that don't reconnect (maybe 1% at most). In earlier versions of SageTV before the metadata was written to the files they would then become imported videos. Now with V7 they are still listed in recordings. They have lost the recorded channel info (that 1%). The description is still present (all 100%). EDL files will still work in imported directories.

Opps forgot to post how I do this.

1. Put edited file on same drive as the imported folder but not in folder that is below the imported folder
2. Stop SageTV
3. Delete original recording
4. Move recording from temporary location to import folder - this should be quick
5. Restart SageTV and let it finish scanning your import folders
6. Once done go to Archived Videos and find your files. If you don't want them in Archived Videos then just toggle the archive flag

Note I have 20 - 30 TB of videos on my unRAID server that are still connected by following the above steps.

Seems to work fine, as you said. It is greatly appreciated. Thank you.

ehfortin