95+% fragmentation on recording drives

[Crashless]

I'm amazed that it worked at all! The defrag should have to restart every time something new is written to the disk, you really shouldn't run defrag with ANYTHING running, much less SageTV.

The only way I can think of doing this would be to hack that plugin that shuts down you system in between recordings...

But then my memory isn't the best, so that could have just been something someone was talking about doing.

Your best bet is to do it manually. I would do it on a slow Saturday afternoon when there wasn't anything worth watching on. Depending on your system, it could take a while.

[aperry]

I came up with a manual process that I run very infrequently (sometimes I get the occasional microskip, and I try this). I have two drives, E: and F:. Here are the steps I do, and I have included some of the XCOPY commands in case anyone's interested.

1. Exit sage (and service if necessary)
2. Copy everything from F:\ to E:\backup\ (xcopy /e/v F:\*.* E:\backup\)
3. Delete everything from F:\
4. Copy everything back from E:\backup\ to F:\ (xcopy /e/v E:\backup\*.* F:\)
5. Delete everything in the E:\backup\ folder
6. Copy everything from E:\ to F:\backup\
7. Delete everything from E:\
8. Copy everything back from F:\backup\ to E:\
9. Delete everything in the F:\backup\ folder

After this is done, all my files are unfragmented. And it takes about 1/10th the time of a true defragment. I DO NOT run this using a batch file as I want to makes sure every step completes successfully.

Thought this might be of some help...

[mayamaniac]

if you have enough free space between two drives to move all files back and forth, would'nt it be faster just to store everything on one drive and leave the other one empty, then you can just do one copy operation (*.*) from one to the other, then go into disk management, and just change swap the letters of the drives. This is faster than doing multiple copy operations back and forth.

[Mortanis]

I had the same problem when I first got Sage setup. It got up to 90+% fragmentation, and was causing heavy stuttering.

I wrote a VBScript than Defrags my drive, and set it as a scheduled task to run every morning at 3AM. Never had a problem since, and it keeps everything fresh.

[Mysticeti]

Follow up question: I read that recording two shows at once onto the same drive is a sure way to jack up your fragmentation. If I have two drives set up for Sage to record to when it comes time to record two shows at once will Sage be smart enough to send the two streams to different drives? I suppose I should just try this and see what happens.

Thanks.

P.S. In my humble opinion one of the goals the Sage team should consider setting for itself is that of making Sage as reliable and maintenance free as the best set top DVR. Sure Sage does much more than most set top DVRs and I don't expect Sage to do absolutely everything perfectly but one has to realize that end user's (and WAF) expectations are gradually rising. If the stock answer to "stuttering caused by fragmentation" is reformatting with a certain cluster size and running defrag periodically then you've just lost a certain segment of your potential user base (or worse, those uninformed users may give Sage poor marks in public reviews). It would be nice if Sage took care of the fragmentation for me like a set top box does but failing that having hooks that allow the user to do certain things (e.g. defrag) when there's a window of a certain size in the recording schedule would be sufficient.

Just my two cents... Not meant to be a dig.

[aperry]

Originally Posted by mayamaniac

if you have enough free space between two drives to move all files back and forth, would'nt it be faster just to store everything on one drive and leave the other one empty, then you can just do one copy operation (*.*) from one to the other, then go into disk management, and just change swap the letters of the drives. This is faster than doing multiple copy operations back and forth.

While that may be true, I have three PVR-500's, and by having two drives, there is less of a chance that I would have six streams going to the one drive at the same time. Plus, there are rare times where I have enough between the two drives that it won't all fit on one drive. Most times though, since I delete stuff as I watch it, there's really not that much I need to copy.

[rfutscher]

With a video file systems it is important to use large blocks. With large blocks it doesn't matter if the drive becomes fragmented. By the time you fill or empty the MPEG buffer for a encode or decode it is time to move the heads anyways to service another stream. With a number of tuner cards and clients there is always another stream to service.

[joe123]

Having multiple tunners fragments drive(s) like mad due the fact that those tunners are writting out the data in small chunks and those chunks are written anywhere they fit.

This is why when do you a copy (or xcopy), you get one continous file with no breaks (fragments).

So, this seems to be easy to fix. Why not simply have SageTV at the end of a recording copy the file to a temp name, delete the original file, and rename the temp to the origninal? Something like this:

After a recording of say "Batman-returns"

copy "Batman-returns" to "Batman-returns-temp"
delete "Batman-returns"
rename "Batman-returns-temp" to "Batman-returns"

Anyone knows the location of where I can put script which will be called after a recording is done?

[mayamaniac]

you can probably setup DirMon2 to run the script or ask JereJones on advice on how to do so or even add it as a features into DirMon2.

[stevech]

Anyone REALLY know if video "stuttering" (too-slow I/O) is significantly better/worse with:

a) Ordinary EIDE drives
vs.
b) USB2 data drives, and the boot drive is EIDE

[blade]

Originally Posted by stevech

Anyone REALLY know if video "stuttering" (too-slow I/O) is significantly better/worse with:

a) Ordinary EIDE drives
vs.
b) USB2 data drives, and the boot drive is EIDE

I don't doubt that some people have problems with stuttering, but I still don't see how it can be caused by fragmentation or any bus speed limitations.

My server is an old BX chipset with the front side and memory bus running at 66 mhz. I have (2) 200 gig, 7200 rpm IDE drives with 8mb cache. Both drives are currently 99% fragmented. I was capturing 4 shows last night at 3.2 gb/hr each, all 4 shows were being processed by ShowAnalyzer in "real time" and I was watching another show on the client. There was no stuttering or anything else that would indicate a problem.

Are the people having problems using onboard controllers or pci cards? I use an ATA controller card due to the BX not supporting the newer large drives. I'm just wondering if some of the onboard ide controllers are poorly implemented. I've had motherboards that had spikes in cpu usage, seemingly long access times, and other problems. None of which I've had when using a controller card. One of my nforce2 boards is horrible when trying to use the Nvidia IDE driver.

[AboveUnrefined]

I've been trying to get a setup where all recordings go onto a ramdisk then flush out to a hard drive to avoid abuse of the hard drive. The more fragmented the drive the more work it's doing and the more work it's doing the more heat it generates and the more heat the quicker it dies... It doesn't seem like anybody likes that school of thought though!

[Jackal24]

Originally Posted by AboveUnrefined

I've been trying to get a setup where all recordings go onto a ramdisk then flush out to a hard drive to avoid abuse of the hard drive. The more fragmented the drive the more work it's doing and the more work it's doing the more heat it generates and the more heat the quicker it dies... It doesn't seem like anybody likes that school of thought though!

This sounds like a good idea with one problem. How big a ramdisk are you gonna need? Wouldn't it be cheaper to just use a cheap HDD. If it dies, just get another one. Gotta be cheaper than several gigs of RAM.

[gplasky]

Originally Posted by AboveUnrefined

I've been trying to get a setup where all recordings go onto a ramdisk then flush out to a hard drive to avoid abuse of the hard drive. The more fragmented the drive the more work it's doing and the more work it's doing the more heat it generates and the more heat the quicker it dies... It doesn't seem like anybody likes that school of thought though!

I just don't know if it is something to worry or workaround. Some of the latest Cheetah drives from Seagate have a MTBF of 1.4 million hours. I've had the same 2 x 160 Seagate SATA drives recording probably 40 hours of TV per week for the last 4 years without a peep. Formatted in 64K I have NEVER defragmented them. No perfornmance issues. I redid my setup once and formatted the drives clean about 6 months ago after updating the BIOS and other upgrades I had been holding off. I've recently built a FreeNAS box with 2 x 400GB Saegates. Other than making sure the cases are adequately cooled I just run them. (And I don't mean jet engines-power supply fan, 2 x 80mm fans out the back. Not too noisy but you can hear them when everything is turned off.) Cables are carefully routed and I make sure air can freely flow through the case.

Gerry

[Naylia]

Originally Posted by blade

I don't doubt that some people have problems with stuttering, but I still don't see how it can be caused by fragmentation or any bus speed limitations.

I have no issues with fragmented SD shows ever. However, HD tv shows that are severly fragmented can give me issues. Just doubled the size of my pvr drive and set to defrag every night as too often it filled up before my defrag got to it, and then the defrag became useless.

[Polypro]

Diskeeper "Set It & Forget It" is what I use on all my drives...I don't defrag my recording drive however.

P

[stevech]

In my system, when the camera pans a scene, there are dropped frames until the panning slows or stops. Like every second or so you see skipped frames.
Some may call this stuttering.

The prior posts argue that it isn't the disk I/O limitations.
So it's either
1. MPEG encoder can't cope - or can't get its stream consumed fast enough (in my case, this is USB2).
2. video card sampler to S-Video.

I wonder if (2) is really possible - as this process would have to be sensitive to the busyness of the scene. I doubt it.

So that leaves (1).

Agree?

[blade]

Originally Posted by stevech

In my system, when the camera pans a scene, there are dropped frames until the panning slows or stops. Like every second or so you see skipped frames.
Some may call this stuttering.

The prior posts argue that it isn't the disk I/O limitations.
So it's either
1. MPEG encoder can't cope - or can't get its stream consumed fast enough (in my case, this is USB2).
2. video card sampler to S-Video.

I wonder if (2) is really possible - as this process would have to be sensitive to the busyness of the scene. I doubt it.

So that leaves (1).

Agree?

Are you actually getting dropped frames or does it just stutter and appear that frames are being dropped? Using too low of a capture rate, weak video card, not enough cpu power, crappy decoder, decoder settings, and network problems can all cause this type of behavior. I'm sure I probably left out a few.

Watch Fox news and look at the ticker. Does it scroll smoothly or does it stutter as it goes across the screen?

[mayamaniac]

Easily way to test if stuttering is caused by fragmentation is to play the video with WMP or other players, and possibly while SageTV is running.