New server build...opinions please

[BigSwifty]

My current system (see sig) is not up to the task of recording 3 steams and trying to watch one at the same time. Very choppy and perpetual spinning circles when trying to FF. I've seen other recent threads discussing single or dual xeons vs the i7-6600. Concerns are the power consumption difference and, more importantly, noise. The box sits in the living room and the server also functions as the primary client.

It seems from what people have opined that the i7 will have enough hp to handle what I need. Thoughts from those who run similar systems or sage sages?

[alfi33]

I guess I'd be a bit surprised if your server CPU was the problem. Based on what you described, unless you have other stuff going on (like transcoding), I'd be more inclined to think that your bottleneck was a disk I/O issue, or maybe even a network issue.

Have you confirmed that the CPU is maxing out? Is the choppiness on both the server and the client?

[BigSwifty]

Thank you. It seems it is the transcoding. Each utilizes 25% cpu cycle. The client/server is the same box.

[BigSwifty]

Some more info.

My server also functions as a local client. I capture the Nextcom streams and it's sent over the network to a box running the storage drives (in a Snapraid array). When it is fully recording and viewing, the network utilization is still very low. So, I don't see how the network is a bottleneck. Trying to timeshift and FF through an active recording is painful. Example, extending the world series for game 1 led to lots of choppiness and spinning arrows. For game 2 I didn't extend but just recorded the show that followed. The original (finished) recording behaved fine. I'm 99.9% sure it's not a HD issue since it occurs on various healthy drives.

The server also pulls comskip duties. I don't run cs on start but when it starts processing that's another 25%. So again, can the i7 handle recording 3 streams while watching 1 and cs processing a file? This is a drag. I don't want to drop the $ on a new cpu and board unless I have some reassurance that an i7 (which one?) can handle what I want it to do.

[alfi33]

Originally Posted by BigSwifty

Some more info.

My server also functions as a local client. I capture the Nextcom streams and it's sent over the network to a box running the storage drives (in a Snapraid array). When it is fully recording and viewing, the network utilization is still very low. So, I don't see how the network is a bottleneck. Trying to timeshift and FF through an active recording is painful. Example, extending the world series for game 1 led to lots of choppiness and spinning arrows. For game 2 I didn't extend but just recorded the show that followed. The original (finished) recording behaved fine. I'm 99.9% sure it's not a HD issue since it occurs on various healthy drives.

The server also pulls comskip duties. I don't run cs on start but when it starts processing that's another 25%. So again, can the i7 handle recording 3 streams while watching 1 and cs processing a file? This is a drag. I don't want to drop the $ on a new cpu and board unless I have some reassurance that an i7 (which one?) can handle what I want it to do.

Recording and even playback usually just isn't all that CPU intensive. So unless you're seeing your CPU utilization really high like 80-100%, then I'd suspect the problem is not your CPU so it wouldn't be fixed with a new CPU. What does the CPU utilization look like on the server when you're experiencing the choppiness? If you see that it is really high, what process is the biggest offender?

Comskip can definitely cause disk thrashing...which can cause choppiness when trying to watch something. Have you tried disabling Comskip processing completely and see if that make the choppiness stop?

To answer your question, yes, any i7 should be able to handle what you're asking...but my little i5-2400 (as well as my old AMD Athlon 64 X2) is able to handle similar duties just fine.

[BigSwifty]

So an update...
Please remember that I am not an IT pro but know just enough to get me in trouble.

I installed the utility Bandwidth Monitor to get some more info. When the issue occurs during timeshifting, the data stream drops considerably from the server. I was thus thinking it's either a network problem or a disk issue. I had an extra switch at work and swapped that in not really that hopeful that it would be so simple as a failing switch. It didn't help. I then narrowed the issue down to occur mainly with 2 of the disks in the server. I don't have any smart errors but can the disks be failing despite this? My disks are getting older and I figured that it couldn't hurt to replace and add more GB at the same time.

I'll be cloning and then swapping them out this weekend; hopefully with some good news to report.

Can we talk about HDs for a bit? I had been using standard desktop drives but, as capacity needs increased, I started using WD Purple drives that are supposedly optimized for multiple video streams. I don't believe the drives involved are any of the purple persuasion. I read that the WD Red and purple drives may be a bit faster. Any comments on possible improved IO performance with these? Hopefully, new HDs will solve my problem.

[JustFred]

Originally Posted by BigSwifty

I had been using standard desktop drives

Given the absence of SMART errors, it's worth looking a bit deeper.

There's a huge range of performance across "desktop" drives. Exactly which disks (mfgr, model) are you using? 5400 vs 7200 rpm? What size are the internal disk buffers?

In general, 5400 RPM disks can perform significantly slower than 7200 RPM drives when it comes to random access. It might not be noticed with a single stream, but could become significant when processing multiple streams and ComSkip.

Originally Posted by BigSwifty

I'll be cloning and then swapping them out this weekend; hopefully with some good news to report.
...
Hopefully, new HDs will solve my problem.

.. or mask the root cause.

Simply swapping out disks will change a whole bunch of things and even if there is a (temporary) improvement, you may end up revisiting this issue a few weeks/months down the road unless you're able to dig a bit deeper now. Not only would there be a hardware change, but Sage will be working with disk(s) that contain an unfragmented filesystem.

A heavily fragmented filesystem and/or low-performance disk(s) can certainly expose problems with multiple streams. Have the disks been recently de-fragged?

What allocation size was used when the disks were formatted? Back in the Sage6 days, Sage recommended that the disk be formatted with 64K clusters. Sage7 supposedly dropped that recommendation, but I've continued to follow that practice.

Before swapping any hardware, it's probably worth running some tests to characterize the performance of the disks themselves, and then your network.

As a simple test, I'd suggest using Windows Explorer to copy a relatively large file (~ 1GB or so; approx the size of one of your recordings) from one disk to a 2nd **local** disk. Note the transfer rate (click 'More Details' in the window that appears as the copy proceeds). Next, rename the destination file and copy it back to the source disk; note the transfer rate again. Now repeat the above 2 copy operations between two folders on the same physical local disk, and note those transfer rates. Finally, repeat the copy operation between two disks across your network. That should give a pretty good idea of both disk and network performance limitations.

For reference, running Win7-x64 on one of my relatively old (Gigabyte GA45-UD3P) motherboards, I routinely see copy speeds of 90-110MBps between two different local disks (reasonably modern 7200RPM SATA-II disks with 64MB buffers). But that rate drops to 65-75MBps when copying between 2 folders on a single 5400 RPM disk (Hitachi 4TB Coolspin, 5900 RPM with 64MB buffer). And copying across my Gigabit network interface gives about the same speeds as local copies. All of the above is with no filesystem fragmentation.

[UgaData]

the WD Red is a NAS drive and spins at 5400 RPM (maybe 5900) and is meant for multiple drives (up to 8) in a NAS. But they work fine as individual drives as well.

The WD Purple is for video surveillance systems and I believe also spins at 5400 RPM.

In most cases the 5400 RPM speed should be fine and will run cooler than the 7200 RPM drives. Yeah, I prefer using 7200 RPM drives most of the time, even though it isn't a necessity.

I format all non OS drives/partitions with 64K blocks (sector size). It can help with fragmentation and I think it is more efficient when storing large files (like video recordings and/or movie rips). You certainly don't have to format with 64K blocks for things to work but it isn't that difficult to do when setting up the drive/partition.

[Rob23]

Have you also confirmed that your server/client's operating system isn't enforcing network or bandwidth throttling? Both Vista and Windows 7 can do this out of the box. I experienced similar issues and changing the throttling settings in the registry fixed the problem for me. Here's a discussion:

http://www.thehtpc.net/htpc-tips-and...k-performance/

[Fuzzy]

NAS storage is likely your limiting factor above anything else. Writing 3 streams to the NAS. Do you have it all going to a single 'share' on your NAS, or does the NAS have a separate share on dedicated to each data disk? Sage has a 'bandwith' mode of storage arrangement that will try not to record to the same recording location more than once at a time if it can, so if you have 3 sources, you can set 3 recording locations dedicated to 3 separate disks, and it will spread out the writing, and not have to constantly seek between writing 3 different files on the same disk.

Of course, local storage to a set of drives will always be better performing than a NAS, if you can do it.

[JustFred]

I didn't see any mention about what other services are being run on the system where the media content is stored, but some of these can have a significant impact on Sage. For example, running a Torrent client on the same disk that contains Sage recordings can cause a significant amount of disk thrashing. This makes it all the more important that the disk(s) perform well for random access disk I/O (think spindle speed, disk buffer size, etc.)

Another factor to consider is the effect of any anti-virus software (I'm assuming that you're running that). It's known that the real-time virus scan which occurs during file transfer can significantly impact recording/viewing. As a simple test, try disabling the A-V software and see if the problems with Sage persist.

In my case, it was easiest to completely exclude the entire disk from any A-V scanning, since it stores only Sage content.

A bit of a digression, but since surveillance (e.g., WD Purple) drives were mentioned: I'd strongly recommend against using these drives for Sage, or for storing a media collection. These drives are designed for a very specific use-case, and that scenario doesn't align well with Sage/media. Their caching and error recovery policies are significantly different than a typical desktop disk. They're optimized to handle a multi-stream write-intensive environment, and intentionally return bad read data under certain use conditions.

The notion that it's acceptable to return occasional bad read data assumes the result will be a momentary playback flicker on your screen (and that bad spot of the surveillance video may never even be watched!) Unfortunately, that's a particularly bad assumption when there will be any further post-processing to the read stream (e.g, the remuxing that likely occurs with ComSkip). It's also a bad assumption when the disk contains a media collection amassed over several years, and then disks age and read errors increase.

Furthermore, a typical desktop disk is designed based on the assumption of a "traditional" 80% read, 20% write distribution. But surveillance disks assume closer to a 5% read, 95% write mix. This 5/95 ratio works very poorly for Sage with multiple simultaneous record/play streams.

And don't even consider storing Sage content using any of recent crop of "archive" drives with their shingled recording...

[BigSwifty]

Thank you everyone for your recommendations...
Anyone wanna buy some slightly used WD Purple drives
As JustFred suggested, I ran some transfer tests and the surveillance drives' speeds are abysmal on my system. Since these 2 drives were the most recent added Sage was writing to them preferentially. Transfer speeds across 7200rpm drives on the server and across the network were similar to what you describe. The WD Red I have is a 7200rpm drive.

Fuzzy, I am aware of the settings for "bandwidth" vs "available space?". I don't think SnapRaid is the limiting factor here. It has multiple data disks and 1+ parity disks for recovery. If I ever have to rebuild I will probably look at unRaid and docker.

I replaced the surveillance drives with WD Black versions, rearranged recordings so that they are the target for essentially all new recordings and all is well again with my system.

Also, for completeness, the storage computer is just a file server running Win7, SnapRaid and AV scans running @3:00am (nothing usually recording).

So for the TLDR version: on my system which was running quite stable for many years, the recent addition of WD Purple drives ground it to a halt. 7200rpm drives for me going forward.