Dream/Reality RAID Fileserver/HD Sage Server

[ToxMox]

Originally Posted by davephan

What OS were you planning to use Linux or Windows? If you will use Windows, what version of Windows? If you are going to use Windows 2003 server, you will need to get the Enterprise version if you want to exceed 4 GB of RAM.

Sometimes (rarely, but it does happen) RAID arrays do fail. How do you plan to backup your videos? It is tough to backup those large drive arrays, unless you have an expensive tape library or another RAID array!


Dave

Will most likely run Windows. Don't plan on exceeding 4 GB of RAM so that isn't a problem. Since it is pretty rare that a RAID fails I'm just going to rely on the RAID to be solid. Right now I have no redundancy so this will be a step up for me.

[ToxMox]

Originally Posted by sniper's_scope

expect to pay $2000-$4,500 for server based hardware barebones, pending on size hdd's and the quantity you put in the negates its price, i'd be expecting $5,000-$15,000 on hard drives as a min price range..

I don't know what kind of drives you think I'm going to put in this machine but as I mentioned in the 1st post that I'll probably start with 4 raid edition 500 gb WD drives at approx $170 ea = $680. By the time I get around to buying the drives I'm hoping they will be closer to $135 ea. As I add new drives after that they'll be cheaper and cheaper.

[ToxMox]

Originally Posted by salsbst

I have the same 16-drive case. I run it with only two of the case fans. I'm able to do this because I didn't load up the drives (I only have 6 drives in the case).

It's nice that with the Areca, you can monitor each drive's temp and figure out a placement that works for your situation.

What troubles me about the Chenbro case is that it stacks the drives so tightly. If you fill the case, you need some serious pull from the fans in order to get air flowing over the drives. The moment you stack one drive on top of the other in that case, you have serious heat issues. Even with the full complement of five case fans, I would expect that you'd have semi-regular drive failures if you load it up. ToxMox, if you have drives that end up next to, but especially on top of each other, you might want to make sure you have a drive or two on hand to replace when needed.

I hope that this isn't the case as I do plan to eventually fill the bays with drives. I'll just have to get more airflow through it or something then. I have to think that the case isn't THAT bad at getting the drives too hot.
I guess time will tell

[sniper's_scope]

if you're going to be playing with a case with alot of space it would stupid mpot to fill it to it to its maximum compasity with hard drives, that also depends on the the compasity of the storage that server barebone can take...

it doesn't take long for the hdd's to fill up, with 16 500gb that would give you 8tb give or take 320gb after formatting, if you got the cash why not build the server with 750gb hdd's that gives you close to 12tb of storage space give or take 320gb pending on the size of server chasis in relation to how many hard drives you can stick in it....

so my last quote stands when it comes to prices when filling these server cases with hard drives...

though if you're planning to use vid capture cards for fta and cable services i would strongly suggest dual or quad cpu based mobo's with alot of ram as 4gb ram will not be enough when encoding from digital transmissions, you'll need the cpu space even using dual core cpu's...
if you are covering multistreams at once a single cpu isn't enough, reading a thoughout many forums i've come across people are building systems on single processor whether it be single or dualcore cpu's in use, they're are finding they are starting to get bottleknecks when it comes processing multiple analog and digital sat & fta tv streams on thier current systems as they are using sometimes 4-5 cards in the system nothing to happen, when they start getting slow down slugish responces through the system they automatically blame the capture card, in some cases an update in drivers can fix minor issues though in some cases it's the ram and cpu at fault which drivers can't fix...

i have the belief that the more capture cards you use the more cpu, ram and hard drives you need to process the downloaded information you get from those capture cards... while some the decode gets done gets done by the capture card, alot still gets done by the ram and cpu..

[ToxMox]

I do plan on filling the case but not until I need the space. It would be a waste of money to buy all the hard drives at once when say I can start with one RAID 5 of 500gb drives then when I need more space which could be a year away I could get 5 1-TB drives for perhaps the same price or cheaper and so on.

if you got the cash why not build the server with 750gb hdd's that gives you close to 12tb of storage space

Who says I've got the cash to do this?

you'll need the cpu space even using dual core cpu's

As I said in the first post that I will likely run two dual core opterons. I don't think it will be as much of a bottleneck as you seem to think it will be as it is my understanding that some capture cards pretty much do the encoding work so HDD write speed would be my bottleneck there not CPU although I'm sure it helps a little to have 4 cores running. I also don't think RAM is as important as you seem to think it is. Is there something I'm not aware of that indicates that tuner cards require enormous amounts of memory to do their job?

[tmiranda]

This is OT but I just have to ask. ToxMox, what do you do for a living and are they taking applications? Your toys are most impressive.

[ToxMox]

I am a graphic designer and system administrator. I also do consulting work on the side (which is what pays for most of my toys usually). To be honest I don't really make that much money compared to all my friends in the same industry but at least I like my job and I wear whatever the hell I want.

The reason I'm getting the toys is that this is pretty much the last year I'll be able to do this sort of a thing as we are planning on starting a family this year and we are finally moving into a house in the next few months so this is kind of my last hurrah and I have a great wife.

[sniper's_scope]

i used the 750gb hard drives as examples only.

for processing multiple streams at once, i doubt a twin dual core cpu mobo would able to process all the information at once without having some type of problem.... i'm refering to realtime decoding, not delayed processing..

[ToxMox]

Are you referring to realtime transcoding to divx? I have no intention of doing that since it isn't at all needed. Or do you just mean the ENcoding of the streams into the MPEG or TS format the cards spit out? If so then the procs should easily handle that.

[sniper's_scope]

no not divx

you have multiple sources recieving at the same time, the cpu wouldn't be able to handle all that information all at once regardless of what you file system you convert it into!

i wasn't strictly speaking of the recording side of things, i was talking about streaming conntent just like you're watching a normal tv...

[ToxMox]

Dude...how much bandwidth do you think an HD stream takes? Why wouldn't my server be able to record 4 HD streams locally then stream 1 maybe two of them to the TVs in my house. I would think one CPU could probably handle this task. I'd love to hear a technical explanation of why you think this wont work.

[ToxMox]

I'll give a little technical info backing my side up:
For example take a Fustion HDTV Gold. The system reqs specify these:
Pentium 3 750MHz or Celeron 900MHz CPU with 128MB Memory (with ATI RADEON series with DXVA VGA, NVIDIA MX440, FX series)
Pentium 4 1.6GHz with 128MB DDR266 or faster memory for non DXVA VGA
Windows ME/ 2000/ XP or later version of Windows
Ultra DMA 66 enabled main board and HDD for recording

Now the P3 750 is an overestimate for recording because the requirement is to be able to playback the file which the server won't be doing it will only stream files over the network. Assuming I needed every drop of a P3 750 to record 1 HD file from the Fusion card how is it that a Dual core 2 GHZ opteron processor wouldn't be able to handle 4 HD streams recorded from say 4 fusion cards? I think the RAID will greatly exceed the req of having UDMA 66 as well.

[bcjenkins]

Without reading too much in your spec list. I think you'll be fine. If it were me, I would worry about disk i/o. (which it sounds like you did) There is no encoding of files taking place, just a capture of streams and sending them back out. Unless your client machines can't access the network locations of the recordings directly. Then the server has to "transcode" them.

B

[sniper's_scope]

think of what you're saying, you just gave me the specs of running just 1 card.

you add 4-5 simular cards into that mix then your system specs go up, cause the manafacturers give min requirement for 1 card only not multiples of the same card which most likely covers anolog, sd and hd source fequency transmissions for fta and cable sources... then you have to add the aspects of recording, server and network usage also comes into the mix, which i doubt most people think of when they build thier htpc...

[bcjenkins]

Originally Posted by sniper's_scope

think of what you're saying, you just gave me the specs of running just 1 card.

you add 4-5 simular cards into that mix then your system specs go up, cause the manafacturers give min requirement for 1 card only not multiples of the same card which most likely covers anolog, sd and hd source fequency transmissions for fta and cable sources... then you have to add the aspects of recording, server and network usage also comes into the mix, which i doubt most people think of when they build thier htpc...

Actually analog channels will require more CPU with these cards than the HD/Digital ones. Analog is done through software encoding. All that is going on is a capture of a stream from the cards through the PCI bus. There is no processing of the stream.

B

EDIT: Oh and the specs on the cards take into account playback as well. It doesn't sound like that is happening on this box.

[blade]

Originally Posted by sniper's_scope

think of what you're saying, you just gave me the specs of running just 1 card.

you add 4-5 simular cards into that mix then your system specs go up, cause the manafacturers give min requirement for 1 card only not multiples of the same card which most likely covers anolog, sd and hd source fequency transmissions for fta and cable sources... then you have to add the aspects of recording, server and network usage also comes into the mix, which i doubt most people think of when they build thier htpc...

No offense, but do you have any experience with these things?

For SD most people use hardware capture cards. My old server was a celeron 366 mhz with 320mb of ram with ATA 100 drives and it had no problems capturing 4 shows at once, streaming to 2 sage clients and running 1 instance of comskip. Software capture of SD is the only thing that would be demanding.

HD streams are already compressed so they're simply written to the disk. I think you're greatly over estimating how much power is required for a server.

[ToxMox]

Originally Posted by sniper's_scope

think of what you're saying, you just gave me the specs of running just 1 card.

you add 4-5 simular cards into that mix then your system specs go up, cause the manafacturers give min requirement for 1 card only not multiples of the same card which most likely covers anolog, sd and hd source fequency transmissions for fta and cable sources... then you have to add the aspects of recording, server and network usage also comes into the mix, which i doubt most people think of when they build thier htpc...

Just a rough estimate but say a single Athlon 64 clocked at 2ghz is roughly 10 times faster than a P3 750. Like mentioned by others those system requirements account for playback and recording at the same time so chances are recording takes less than half of that 750 mhz of the P3 and that is being generous to record one stream off the Fusion tuner. So how would 2 dual core opterons let alone just one not easily be able to handle the CPU load of recording 4 HD streams and also another say 6 SD streams via hardware encoder cards?
I feel like you are being vague and just guessing at how much CPU and load it would take to run this kind of a setup. As far as I'm concerned I'm going a little overboard on the CPUs and specs I plan on building into this server for what it will be doing.

[ToxMox]

Take for instance jptaz's specs since he posted earlier in this thread that he is running 3 HD tuners as well as 4 SD tuners. He is running this on and Athlon XP 3200+ with 1GB of RAM without issue AND the same machine is doing playback at 720p to his TV. I have all the confidence in the world that I shouldn't have any problems.
Not to mention the additional load on the server from streaming to 2 MVPs which strains the server's CPU.

[korben_dallas]

You are choosing RAID 5/6 because of fault tolerance, correct?

Because it is not necessarily better performance. If you are wanting performance, then I'd be inclined to look into JBOD if I were you. Alot of RAID cards will do JBOD now, and if a drive crashes, you only lose the files on that drive (not the whole RAID volume).

[ToxMox]

Originally Posted by korben_dallas

You are choosing RAID 5/6 because of fault tolerance, correct?

Correct...a nice balance of fault tolerance with usage of space.