Plugin: Sage Artifact Delete (SAD)

[voorhees]

Originally Posted by electronics4lif

I looks like I am also experiencing this bug. SAD.log says that all of my 'test' files are not media objects. I am subscribed to see if something gets resolved with this bug.

I submitted a bug report yesterday. They jumped right on it and requested several things (verify files are good in SageTV, command used in Expression Evaluator, and Wiz.bin file). I got them the data when I got home from work today, so hopefully they will be off looking at it tomorrow. If you have seen the same problem, I do not think it would hurt to let them know so they know this is not an isolated case.

Just for curiosity sake, are you running v7 and any HD200s? Have you seen issues with the HD200 turning off ~3 sec after you turn it on? Have you gotten Playback Exception errors when trying to play any media files from SageTV on the HD200? Or, issues like these, these, or these. I am not sure how, but I am interested to see if the non-registered media object thing is related to the issues several folks (including me) are having with the new beta firmware on the HD200.

[Narflex]

Originally Posted by voorhees

I submitted a bug report yesterday. They jumped right on it and requested several things (verify files are good in SageTV, command used in Expression Evaluator, and Wiz.bin file). I got them the data when I got home from work today, so hopefully they will be off looking at it tomorrow. If you have seen the same problem, I do not think it would hurt to let them know so they know this is not an isolated case.

Just for curiosity sake, are you running v7 and any HD200s? Have you seen issues with the HD200 turning off ~3 sec after you turn it on? Have you gotten Playback Exception errors when trying to play any media files from SageTV on the HD200? Or, issues like these, these, or these. I am not sure how, but I am interested to see if the non-registered media object thing is related to the issues several folks (including me) are having with the new beta firmware on the HD200.

This is not a bug in the API. We looked at his Wiz.bin file and the files were recorded to a UNC path; which is why they don't show up when searched for by the D:\ path that was being tested against.

[Slugger]

Originally Posted by voorhees

I submitted a bug report yesterday. They jumped right on it and requested several things (verify files are good in SageTV, command used in Expression Evaluator, and Wiz.bin file). I got them the data when I got home from work today, so hopefully they will be off looking at it tomorrow. If you have seen the same problem, I do not think it would hurt to let them know so they know this is not an isolated case.

Just for curiosity sake, are you running v7 and any HD200s? Have you seen issues with the HD200 turning off ~3 sec after you turn it on? Have you gotten Playback Exception errors when trying to play any media files from SageTV on the HD200? Or, issues like these, these, or these. I am not sure how, but I am interested to see if the non-registered media object thing is related to the issues several folks (including me) are having with the new beta firmware on the HD200.

Originally Posted by Narflex

This is not a bug in the API. We looked at his Wiz.bin file and the files were recorded to a UNC path; which is why they don't show up when searched for by the D:\ path that was being tested against.

That makes perfect sense. Thanks for the update, Narflex. The other issue - SAD incorrectly deleting artifacts for unregistered, but not deleted media files - has already been fixed in the latest version available in the repository. So unless someone reports something new then all should be good.

[voorhees]

Originally Posted by Narflex

This is not a bug in the API. We looked at his Wiz.bin file and the files were recorded to a UNC path; which is why they don't show up when searched for by the D:\ path that was being tested against.

Narflex, thanks so much for looking into the issue. Great to hear it is not a bug.

I thought I had also tried testing it against the UNC path and got the same "null" result, as I mentioned in one of my e-mails to support, but I guess I got the syntax wrong. What would be the syntax to test it against the UNC path?

Thanks.

[electronics4lif]

Originally Posted by Narflex

This is not a bug in the API. We looked at his Wiz.bin file and the files were recorded to a UNC path; which is why they don't show up when searched for by the D:\ path that was being tested against.

I didn't even think of that. I am indeed also using UNC paths. Thank you for tracking down our testing error.

[voorhees]

Originally Posted by Narflex

This is not a bug in the API. We looked at his Wiz.bin file and the files were recorded to a UNC path; which is why they don't show up when searched for by the D:\ path that was being tested against.

Yep, that was it. Use \\\\localhost instead of D:\ and I get data instead of "null". It looks like I tried \\localhost instead of \\\\localhost; missed the extra two backslashes. Thanks again for tracking this down.

[Slugger]

I have unsubscribed from this thread. "Quick(ish) response" support for this plugin is no more. I may or may not read further posts in this thread and if I do read them then I may or may not respond to them. More details here.

[waylo]

Hi everyone,

I have the plugin Sage Artifact Delete installed which is supposed to "listen for media object deletion events and will delete any artifacts associated with the media object automatically."

It's been a long time since it was created. Has its functionality ceased? In v7 and/or v9?

I was hoping it would supplement the cleanup function of the Commercial Detector plugin for my .edl files. That plugin is able to erase shows' .edl and other comskip associated files that are from recorded favorites which are subsequently manually deleted. It does NOT remove leftovers from live watched TV. Those video files are auto deleted once you stop watching the channel, but the artifact files are left over. A manual scan for leftovers within the plugin configuration is required every so often to get rid of those files.

* merged *

[tmiranda]

Originally Posted by waylo

Hi everyone,

I have the plugin Sage Artifact Delete installed which is supposed to "listen for media object deletion events and will delete any artifacts associated with the media object automatically."

It's been a long time since it was created. Has its functionality ceased? In v7 and/or v9?

I was hoping it would supplement the cleanup function of the Commercial Detector plugin for my .edl files. That plugin is able to erase shows' .edl and other comskip associated files that are from recorded favorites which are subsequently manually deleted. It does NOT remove leftovers from live watched TV. Those video files are auto deleted once you stop watching the channel, but the artifact files are left over. A manual scan for leftovers within the plugin configuration is required every so often to get rid of those files.

* merged *

CommercialDetector should delete the files from live TV. Please reproduce the problem and post a logfile for me to look at.

[waylo]

Originally Posted by tmiranda

CommercialDetector should delete the files from live TV. Please reproduce the problem and post a logfile for me to look at.

Thanks for the thread merge mod. Couldn't find it in the search, probably cause the thread is 2 years old!

Thanks @tmiranda for chiming in, I'll see if I can grab the right logs. Would it be in sagetv_0.txt? Can you clue me in on what type of entries to look for?

EDIT: Would help if I turned on the debug level to something > none, haha.

[waylo]

Originally Posted by tmiranda

CommercialDetector should delete the files from live TV. Please reproduce the problem and post a logfile for me to look at.

Hi tmiranda,

Thanks for your assistance in tracking this down.

Attached is the relevant log of a live TV recording which was manually ended, initiating the media cleanup. The leftover file was the logo.txt file associated with the recording.

A few things I think I'm seeing which I hope you can weigh in on:

1. The only leftover files in the recording directory (in this example, I'll have to look at more) was the logo.txt files from comskip. It looks like they might not be properly deleted by comskip when comskip stops, if it is in the middle of file scanning. I do have logo.txt deletion turned on in my comskip.ini.

Does the plugin automatically look for logo.txt files? I see in the log it does, but logo.txt is not directly set in my cleanup file extension list.

Though the log shows it was running the processes looking to delete the logo.txt file, it does not look like it actually deleted it.

2. Perhaps related, the Commercial Detector plugin does not like cleanup extensions with a '.' in them. Specifically, with OpenDCT's subtitle creation support, which creates CC1.srt and CC2.srt files, this causes some issues when using the orphan detector. I added them to my extension list which results in the following behavior. (I've actually turned off CC creation in OpenDCT which is why the files are not actually found and deleted).

Let's call our sample recording videofile.ts. Because of the extra '.' in the extension, the orphan detector looks for a parent videofile.CC1.ts file, or videofile.CC2.ts file. Of course it won't find these misnamed parent files, so it will mark and delete the CC1.srt file. For ALL recordings everywhere.

[tmiranda]

Originally Posted by waylo

Hi tmiranda,

Thanks for your assistance in tracking this down.

Attached is the relevant log of a live TV recording which was manually ended, initiating the media cleanup. The leftover file was the logo.txt file associated with the recording.

A few things I think I'm seeing which I hope you can weigh in on:

1. The only leftover files in the recording directory (in this example, I'll have to look at more) was the logo.txt files from comskip. It looks like they might not be properly deleted by comskip when comskip stops, if it is in the middle of file scanning. I do have logo.txt deletion turned on in my comskip.ini.

Does the plugin automatically look for logo.txt files? I see in the log it does, but logo.txt is not directly set in my cleanup file extension list.

Though the log shows it was running the processes looking to delete the logo.txt file, it does not look like it actually deleted it.

2. Perhaps related, the Commercial Detector plugin does not like cleanup extensions with a '.' in them. Specifically, with OpenDCT's subtitle creation support, which creates CC1.srt and CC2.srt files, this causes some issues when using the orphan detector. I added them to my extension list which results in the following behavior. (I've actually turned off CC creation in OpenDCT which is why the files are not actually found and deleted).

Let's call our sample recording videofile.ts. Because of the extra '.' in the extension, the orphan detector looks for a parent videofile.CC1.ts file, or videofile.CC2.ts file. Of course it won't find these misnamed parent files, so it will mark and delete the CC1.srt file. For ALL recordings everywhere.

Something is wrong with the logfile, I didn't find a single CommercialDetector log entry. (They all start with CD Either you posted the wrong thing or I need another cup of coffee before looking at the file

Do you have "txt" in your list of extensions to clean up? If not then add it. Another possibility is that comskip still has the logo.txt file open and CommercialDetector can't delete it.

I'm confused with your description of the .srt files. I don't use them, Please explain how OpenDCT names the .srt file. Your post makes it sound like they are all named the same

[waylo]

Originally Posted by tmiranda

Something is wrong with the logfile, I didn't find a single CommercialDetector log entry. (They all start with CD Either you posted the wrong thing or I need another cup of coffee before looking at the file

Do you have "txt" in your list of extensions to clean up? If not then add it. Another possibility is that comskip still has the logo.txt file open and CommercialDetector can't delete it.

I'm confused with your description of the .srt files. I don't use them, Please explain how OpenDCT names the .srt file. Your post makes it sound like they are all named the same

Hmm, could you take another look a the log? It starts right after the live watching is stopped. I think almost every single entry starts with "CD: CommercialDetectorMediaFile."!

I do have .txt in the list of extensions.

Good point about comskip possibly still having an open file, thus leaving it unable to be deleted! Maybe a delete delay can be implemented?

Here's the relevant release note for when CCExtractor was added to OpenDCT.
https://forums.sagetv.com/forums/sho...&postcount=770

When turned on, it will make 1 or 2 closed captioning .srt files, with the extension CC1.srt and optionally CC2.srt (sometimes there are 2 caption streams). So "videofile.ts" gets "videofile.ts.CC1.srt" and "videofile.ts.CC2.srt" created.

When orphan detecting, the "CC1.srt" and "CC2.srt" extensions cause the plugin to assume the parent file is named both "videofile.CC1.ts" and "videofile.CC2.ts". These files don't exist obviously.

[tmiranda]

Originally Posted by waylo

Hmm, could you take another look a the log? It starts right after the live watching is stopped. I think almost every single entry starts with "CD: CommercialDetectorMediaFile."!

I do have .txt in the list of extensions.

Good point about comskip possibly still having an open file, thus leaving it unable to be deleted! Maybe a delete delay can be implemented?

Here's the relevant release note for when CCExtractor was added to OpenDCT.
https://forums.sagetv.com/forums/sho...&postcount=770

When turned on, it will make 1 or 2 closed captioning .srt files, with the extension CC1.srt and optionally CC2.srt (sometimes there are 2 caption streams). So "videofile.ts" gets "videofile.ts.CC1.srt" and "videofile.ts.CC2.srt" created.

When orphan detecting, the "CC1.srt" and "CC2.srt" extensions cause the plugin to assume the parent file is named both "videofile.CC1.ts" and "videofile.CC2.ts". These files don't exist obviously.

I drank more coffee and foudn the right logfile

First, I can see why it's not handling the .srt files. I'll have to put in some extra code for that.

I'm pretty sure the other files are not getting deleted because they are still opened by comskip. If you look for "Attempting to delete ...." the next line should say "File deleted ...". If it doesn't it means the delete failed and the file has been added to the "deleteOnExit()" queue. Theoretically they should be deleted when the Sage JVM shuts down.

If I can find some time I'll see if I can update the code.

[waylo]

Thanks for putting in the code updates when you have time.

Quick question about the deleteonexit queue and more generally, SageTV upon exit--do you know what the behavior is of SageTV upon a computer reboot?

I have my HTPC set to reboot every several days in the middle of the night using a scheduled 'shutdown /r' DOS command. I would hope SageTV exits properly under this circumstance.

I found some more .edl files left orphaned over the past several days. I couldn't tell if any were scheduled shows, though many were definitely live viewing leftovers. My logs had since elapsed so I couldn't track down what was happening when those parent files were deleted.

[tmiranda]

Originally Posted by waylo

Thanks for putting in the code updates when you have time.

Quick question about the deleteonexit queue and more generally, SageTV upon exit--do you know what the behavior is of SageTV upon a computer reboot?

I have my HTPC set to reboot every several days in the middle of the night using a scheduled 'shutdown /r' DOS command. I would hope SageTV exits properly under this circumstance.

I found some more .edl files left orphaned over the past several days. I couldn't tell if any were scheduled shows, though many were definitely live viewing leftovers. My logs had since elapsed so I couldn't track down what was happening when those parent files were deleted.

deleteOnExit() is a Java method. If you do a shutdown /r the JVM should exit gracefully and should delete all files in the deleteOnExit() queue.

[tmiranda]

Originally Posted by waylo

Thanks for putting in the code updates when you have time.

Quick question about the deleteonexit queue and more generally, SageTV upon exit--do you know what the behavior is of SageTV upon a computer reboot?

I have my HTPC set to reboot every several days in the middle of the night using a scheduled 'shutdown /r' DOS command. I would hope SageTV exits properly under this circumstance.

I found some more .edl files left orphaned over the past several days. I couldn't tell if any were scheduled shows, though many were definitely live viewing leftovers. My logs had since elapsed so I couldn't track down what was happening when those parent files were deleted.

Attached is a new jar file that probably won't fix the issues you are seeing but it will give me more debug data to look at Just stop Sage and replace the existing CommercialDetector.jar with this one. (Backup the old jar in case you need to go back.)

When you notice it does not delete a file it should (including the CC files), post the logfile for me to look at.

[waylo]

Thanks for the debug opportunity. Installed it.

I came to a realization as to why I'm having leftover .edl files. User error.

(In editing the plugin configuration these past several days, I accidentally cut off the 'e' in ".edl", so the plugin was looking for ".dl" files.)

But the logo.txt files still remain. I'll also turn on the CC file creation so we can watch this in action.

[waylo]

Originally Posted by tmiranda

Attached is a new jar file that probably won't fix the issues you are seeing but it will give me more debug data to look at Just stop Sage and replace the existing CommercialDetector.jar with this one. (Backup the old jar in case you need to go back.)

When you notice it does not delete a file it should (including the CC files), post the logfile for me to look at.

Okay, I've had some time to look through and log a few situations.

Sample 1. Sometimes the plugin is unable to delete a cc file and saves it for the exit as you stated. When I've caught this, exiting SageTV normally does delete the file. I have logs of when the live view ends, and of SageTV's user-initiated exit.

Sample 2. This is the log when the shutdown /r script runs. I think Sage does exit gracefully in this example.

So far so good. But here:
Sample 3. Somehow, after a week or so, there were several left over cc1.srt and cc2.srt files. I have to guess these were situations where maybe Sage did not exit so gracefully. When I run the SCAN for orphans option in the plugin, it deletes ALL cc1.srt and cc2.srt files, even for video files still existing. So there are no more cc.srt files anywhere. An example video file in the log where the cc.srt files were deleted but the video file still exists is "AnthonyBourdainPartsUnknown-S09E06-Oman-22913012-0"

(I will keep looking for better logs demonstrating exactly the time when a file does not get deleted when it should. I believe they are mostly if not all cc.srt files).