my wiz.bin and all my recordings are screwed

[Nate]

My wife is out of town for two weeks. We only had a 160gb drive and it wasn't going to hold all of the recordings that would collect when she was gone.

She has been collecting a series that is all about traveling to Europe. She has 42 half hour episodes totaling about 40gb. I moved all of these files across the network to another machine that has the space. I thought this would free up enough space so she wouldn't miss any shows. However, after about a week I realized there still wasn't enough space.

I decide to add a 250gb drive. I shutdown Sage, moved all the files to the new drive, started Sage, and pointed it to the drive. "Sage Recordings" didn't seem to pick up any of the files, so I restarted Sage. Now only 13 recordings show up.

Now that I have searched and read some threads on this forum, I have found out that the link between airing info from the EPG (electronic programming guide) and the mpg file are stored in the wiz.bin file. Sage doesn't keep around all the EPG airing info it has ever downloaded, it only keeps around EPG airing info for the recordings you currently have. So what happens is that if you move some files out of Sage's videos directory, then Sage silently deletes all the airing info from the wiz.bin file. If you later put the files back, then Sage doesn't have any airing info about them and they don't show up in "Sage Recordings". Instead they should up under "Imported Videos" as regular video files.

Maybe it was naive to think I could just move files around. However, Sage never warned me that there were videos it could not find. It never told me it was going to delete the airing info for those videos. It never told me that if the videos ever do come back that Sage will not recognize them as shows that were recorded, only as video files.

Now, you need to understand that I have 137 recordings totaling 153gb. Sage does not recognize 124 of these recordings. Once my wife watches all these, Sage is going to record them again because it doesn't know she watched them. This is not just a waste of time for Sage, but an ENORMOUS headache for my wife, which means it will be just as bad if not worse for me. Months and months from now, Sage will still be recording shows that my wife has already watched. This is going to drag on for a long time, being a continual annoyance.

What started out as something nice I was doing for my wife has turned into a nightmare. I'm sure she would have rather missed a few shows. She comes back on 1/28/07.

I want to fix this. That this even happened is absolute BULL*expletive deleted*. I had no clue I needed to backup my wiz.bin before moving files around. That Sage deleted the EPG information for the recordings it could not longer find is ABSOLUTELY *expletive deleted*. Is this fixable at all? How would I do it? Do I need to look up 124 airings by hand? Some of them are a few months old I would bet.

I have originally purchased v3 I think and I just recently upgraded to v6. It doesn't make me feel better that I just paid to upgrade software that totally screwed me over.

[Opus4]

Do you have a backup of wiz.bin from shortly before you moved the recordings? If so, you can use that & it should still have the show info for the ones that were removed, and it should have EPG info up to almost the current day. You might miss some EPG data for recent recordings, but the old ones would have the data.

- Andy

[nielm]

Originally Posted by Nate

Sage doesn't keep around all the EPG airing info it has ever downloaded, it only keeps around EPG airing info for the recordings you currently have.

And for shows marked with 'watched' or 'don't like' flags.

So what happens is that if you move some files out of Sage's videos directory, then Sage silently deletes all the airing info from the wiz.bin file.

because it thinks the file has been deleted outside of Sage (explorer)


Opus has already mentioned the wiz.bin backup. If you have not made one yourself, Sage keeps one backup of wiz.bin (wiz.bak) from the last time the EPG was updated (which will be within 24 hrs). If you are quick enough, you may be able to use that.

Otherwise you will have a copy of the wiz.bin from when you installed V6 (Wiz.bin.SageTV V6.0.xx.xxxx).

to use either of these:

• make a backup of your current wiz.bin and wiz.bak, and wiz.bin.SageTV V6.0.xxxxx (incase you don't like the results from using the others, and in case sage overwrites wiz.bak on shutdown)
• shut down the sage server/service
• rename or move wiz.bin to wiz.nasty.messed.up
• copy the backup that you want to use and rename it wiz.bin
• restart Sage.

Note that recordings made since the backup wiz.xxx was made MAY NOT be recognised, and will go into imported videos. If the Airings for these new recordings are in the DB, you may be able to re-associate the recordign to the airing by renaming it (see how in this thread)

If you don't like the results, revert back to the previous messed up wiz.bin using the above instructions.


I am sorry that you found out the hard way that moving recordings is ... tricky... There is an entire FAQ section on moving recordings for future reference.

[Nate]

I don't understand why Sage doesn't keep a secondary database of recordings that have just "disappeared". Why does it take the irreversible action of deleting the airing data? We are talking about mapping the recording file name to some airing ID right? Even if it were a bit more data than that, the secondary database could easily scale to millions of entries before becoming any significant size on disk. Make a setting that turns off the secondary database if you want. When unknown files are found in the videos directories, look in the secondary database and restore the airing data.

Better yet, why not allow Sage to query the EPG service with the airing ID to look up the airing data? The airing ID is in the file name right? There has to be some web service that provides EPG data. It could easily have available all EPG data, ever, and allow the data to be queried.

Or how about a setting to make backups of the last X wiz.bin files when recording files mysteriously disappear? Or how about just a warning before it happens? Some education for the user of the result when video files are moved outside of Sage? Anything at all besides just deleting the EPG data forever! It is such a simple thing to want to do. You get more storage or you want to rearrange some drives Sage uses. It is very easy to think that Sage just needs the files in the videos directory and that everything will be fine if you just move them to a different place. As nice as the FAQ is, it isn't obvious that I need to go to the Sage forums and read up first.

The problem is it takes the irreversible action without telling me. The result is the catastrophe that I have right now. I can't believe it would just throw away my data and leave me with nothing! There can't possibly be a reason why this has not been fixed??



I tried wiz.bak but it was the same as wiz.bin. The next newest wiz.bin backup I have is from 12/29/2006 when I upgraded to v6. Using that, Sage recognizes 57 of my 134 recordings. It seems all the newer recordings are not recognized. How can I restore the other 80 recordings so they appear under "SageTV Recordings"? I looked in the thread you linked but it mentioned the problem in the first post. I'll look harder but I was hoping for something pretty explicit.

BTW, thanks for the replies.

[nielm]

Originally Posted by Nate

Better yet, why not allow Sage to query the EPG service with the airing ID to look up the airing data? The airing ID is in the file name right?

yes, but the airing ID is internal to your individual wiz.bin. Someone else's Sage with the same configuration would have different airing ID's.

How can I restore the other 80 recordings so they appear under "SageTV Recordings"? I looked in the thread you linked but it mentioned the problem in the first post. I'll look harder but I was hoping for something pretty explicit.

If the airings are no longer in the DB, then there is no way to magically recover them.

You can use my webserver's Edit Show Info function (bottom of the detailed info page) to add show/episode info for each of these files that are in the Imported Videos, and this will put them into Archived Recordings (which you can optionally display in Sage Recordings), but this has the problems of being:
a) tedious.
b) the EPG ID's generated by the webserver will not match the EPG IDs used by the EPG service, so when the shows are repeated, sage will not recognise that you have already recorded and watched these shows, and will re-record them.

[Opus4]

BTW: I forgot to mention this... after reverting to an older wiz.bin, see this FAQ for a properties file edit that might need to be done regarding properties with 'server_update_id' in the name so that the EPG data doesn't have a problem updating.

- Andy

[FidgetyRat]

I know it is silly that the wiz.bin is this corruptable and crappy. It has been like this for a very long time with no end in sight.


What I decided to do after my first corruption was to use windows XP's built-in backup utility. I tell it to back-up my wiz.bin and my sage.properties file 4am every 2 days.

I still need to get these copies OFF the main hard disks, but haven't done that yet I bet my first HDD crash will inspire me for the next time.



I wish it were more clear in the manual to protect these files with your life, and especially that sage will delete information like this. Some utility to add files back into sage has been needed since v2.

[GKusnick]

Originally Posted by Nate

I don't understand why Sage doesn't keep a secondary database of recordings that have just "disappeared". Why does it take the irreversible action of deleting the airing data? We are talking about mapping the recording file name to some airing ID right? Even if it were a bit more data than that, the secondary database could easily scale to millions of entries before becoming any significant size on disk. Make a setting that turns off the secondary database if you want. When unknown files are found in the videos directories, look in the secondary database and restore the airing data.

You can implement something like this yourself by using Niel's XML generator to export EPG data for each recording to an external XML file, and then running a nightly script to copy the XML files to somewhere safe. Then if disaster strikes, you'll at least have a source of data to enter back into Niel's webserver. If you're clever with script, this data re-entry could probably be automated to some degree.

This doesn't help you recover the data you've already lost, but it's a way to protect yourself going forward.

Originally Posted by FidgetyRat

I know it is silly that the wiz.bin is this corruptable and crappy. It has been like this for a very long time with no end in sight.

Actually wiz.bin has become a lot more stable and reliable in versions 5 and 6. Wiz.bin corruption was common in versions 3 and 4. It's much less common now. And to be fair, Nate's problem is not a case of wiz.bin corruption as such. The program did what it was designed to do and his wiz.bin is well-formed and contains what it's supposed to contain after removing large numbers of recordings. He just wasn't expecting EPG data to be purged as a result of the removal.

In my view the bug here (or one of them anyway) is that wiz.bin is buried deep under Program Files, where people forget to back it up. User data and config files (like wiz.bin and sage.properties) belong under Documents and Settings, where they're more likely to get backed up regularly.

Regular backups of all your important data (including wiz.bin) are an essential part of good computing practices. No matter how bug-free and reliable your software is, your data is still vulnerable to hardware failure and operator error, so backups are still necessary. Never keep just one copy of anything you care about.

[joe123]

Originally Posted by Nate

What started out as something nice I was doing for my wife has turned into a nightmare. I'm sure she would have rather missed a few shows. She comes back on 1/28/07.

You are not alone Mate! This has happen to so many people is not even funny.

Also, when people here tell you that a backup would help/fix this, that's a lie and pure BS. Let me explain.

If you are like most here, you have 2 or more disks for recordings. I have 6.

By the time you notice a FAILED drive, buy a new one, and extract whatever you can out of the old drive, several days have gone by.

Now, sure you can get a backup of your wiz.bin file and restore it and guess what happens now, *ALL* of your new recordings are lost into the imported section.

The entire process with wiz.bin is faulty and to make it worse, it's in binary format when it should have been text so that it would be easier to fix/understand.

[GKusnick]

Losing a few days of data is better than losing a month or a year of data, right? So having a recent backup does help, in the sense that it's better than not having a backup. Backups may not be the whole solution, but that's no reason not to make them.

Should wiz.bin be text? You can argue it either way. Performance issues aside, my feeling is that allowing users to edit wiz.bin directly would cause a lot more problems than it would solve. That said, you can extract from wiz.bin an up-to-date textual copy of the EPG data for your recordings any time you like using Niel's XML generator as I suggested above. I believe there are several other plugins as well for importing/exporting textual information in various formats.

I'm not saying wiz.bin is perfect the way it is. No doubt there's room for improvement. What I'm saying is that leaving your data unprotected while waiting for the Sage devs to fix it is not the only option. There are tools we can use right now to safeguard our wiz.bin data, or at least limit the scope of a disaster, and it would be foolish not to use them.

[joe123]

Nobody is going to argue against having backups and win. Backups are always needed.

As for what is worse, 1, 2, x years lost verses a day, a week of recordings, of course, you and I will take the 1 week of lost recordings over years of lost recordings.

Having said that, I will argue and continue to argue that neither option need be. The design as it stands is flawed, severly flawed. Instead of providing Google videos in version V6 as a new feature, fix this darm problem or give us tools to help us recoup the missing data.

[nielm]

Originally Posted by FidgetyRat

I know it is silly that the wiz.bin is this corruptable and crappy. It has been like this for a very long time with no end in sight.

Can I just make the point that Nates wiz.bin did not get corrupted: it was working as designed, and removed data referring to files that it believed no longer existed on disk...

Nate's issue is not a problem with Wiz.bin, but with Sage's fragility in respect of moving recordings between recording DIRs...

However, the importance of backups cannot be understated: If you consider your sage setup (faves etc) and viewing history important, then backup the wiz.bin regularly. Hard disks do not work forever!

(/me also incrementally backs up wiz.bin and sage.properties)

[nielm]

Originally Posted by joe123

Give us tools to help us recoup the missing data.

Studio gives full access to the wiz.bin DB.

You can re-create the DB info for 'lost' recordings using the studio API -- if you have the data available.

[Ken C]

The recordings are still available, just not showing up. Isn't there some way to correct that ? I realize all the EPG info is lost but as long as the actual recordings are on the hard drive there should be a way to 're-introduce' them into Sage. Could they be moved to the video import directory ?

It would be nice to have the ability to edit the stored info, maybe some sort of manual input. I've got several shows with mangled names it would be nice to correct.

Ken C

[nielm]

Originally Posted by Ken C

The recordings are still available, just not showing up.

If the files are on the hard disk in a recordings directory then they will be in either:

• SageTV Recordings,
• Media Center/Archived Sage Recordings (If you have set the archived flag),
• Schedule Recordings/Extra: View Partial Recordings (If sage considers them a as a non-full recording)
• or Media Center/Imported Videos (if they have lost their EPG info).

Also Check your filter settings for the Sage Recordings view(s).

Originally Posted by Ken C

It would be nice to have the ability to edit the stored info

At the risk of repeating myself

Originally Posted by nielm

You can use my webserver's Edit Show Info function (bottom of the detailed info page) to add show/episode info for each of these files that are in the Imported Videos, and this will put them into Archived Recordings (which you can optionally display in Sage Recordings)

[Ken C]

Nate's problem has been discussed before, the fragility of wiz.bin, the difficult, almost impossible task of trying to add/correct entries, problems with moving recordings. He's going to lose a bunch of WAF as a result. He said in the original post that Sage doesn't recognize 124 recordings. Here we are at version 6 and still no built-in method to deal with these problems.

Is this a common problem with all the software based pvr's ? Anyone have experience with MCE, Beyond, or GB ?

Nielm, I looked at Studio once and realized it's not something I should be messing with. As Harry Callahan said, "A man has got to know his limitations"

Ken C

[joe123]

Originally Posted by Ken C

Nielm, I looked at Studio once and realized it's not something I should be messing with. As Harry Callahan said, "A man has got to know his limitations"

Ken C

Amen! That is way too complicated for the average user. That is what tools are for and there are NONE for this.

Let me re-stated what I said on another post, I followed *ALL* the steps correctly and still lost about 30% of my recordings.

I recently lost 1 of my 6 drives. I stopped Sage, pull the drive out, restarted Sage and went on to retrieve what I could out of the old drive. I was able to retrieve pretty much everything. By the time I was finished, a week had gone by.

I shutdown Sage, installed the new drive with recordings back in the same location as before, everything excatly the way it was, brought SageTV up and guess what happened, 70% of my recordings were OK - In SageTV's menu while about 30% went into the Imported section. File names were not corrupted. Very frustrating.

[Opus4]

For everyone following this topic... a future version of SageTV will be more conservative in terms of removing data for older airings. Right now, airing details are retained for items that have a reason to be tracked, such as being watched, marked Don't Like, and so on. In the future, the fact that an airing was recorded will also cause SageTV to retain that airing's details in its database so that that the info won't be removed when the file disappears. As long as the filename doesn't change, the ID that is part of the filename should allow restored recordings to be matched up with their airing info.

- Andy

[hemicuda]

not trying to fuel the fire here... but I think I remember the timestamp on a video causing problems for me on a test machine last year. If memory serves I had shut down sage completely (yes, the service too) and was doing something with the file that ended up changing the date. After that sage placed it under imported videos. I'm open to comment if my experience was way off base here. I've even tried the touch.exe app that someone on the board was kind enough to suggest.

[Nate]

Originally Posted by Ken C

Nate's problem has been discussed before, the fragility of wiz.bin, the difficult, almost impossible task of trying to add/correct entries, problems with moving recordings. He's going to lose a bunch of WAF as a result.

Exactly. I can't wait to explain where 80 of her recordings are, under Imported Videos, and that Sage is going to record them again if/when they air again.

I think everyone agrees that Sage should not be so fragile when recordings go missing briefly. Andy says it will be fixed in a future version. While that is a bit vague, hopefully it will be fixed soon (6.0.x?).