XXL Sage Install

[digitalgm]

Subject: XXL Sage Install

First let me thank you all for the amount of help I have recieved from this board - you guys are truly amazing and I could not have completed my first setup without your help.

That said, I'll get right to the point -- I am in the process of building our dream home and I would like to include a large SageTV installation. The wife is in love with our current Sage installation and I feel pretty sure that if it grew longer hair and developed a latin accent that she would surely leave me for it. With such a great experience so far, I want to include a MUCH larger system our house so I'd like to spec out what I need and get started. Just so you know, each room has multiple drops of CAT6 and I have even created something of a "server room" in the house (I'm a SUPER-Geek). In any case, please take a moment to look over my system plan and answer some of the questions that I have. Thank you all in advance.

-Regards, AdamR

Requirements:
* Support for 20+ Clients (Including wired HTPC clients, MVP/ME clients as well as wireless clients)

* Provisions for at least 40 Analog Tuners
* Provisions for at least 10 OTA HD Tuners
* Provisions for at least 3 SAT HD Tuners


Proposal:

1 x Master Sage Server
Description: The master sage server would be in charge of retrieving and distributing EPG as well as brokering connections to the media housed in the network encoders

Specs:
- Single P4 3.2GHz / 2GB RAM (More?) / Gigabit NIC



1 x Master Commercial Skipping Server
Description: This machine would be dedicated to performing the commercial skipping process on all network encoders.
Specs:
- Dual Xeon @ 3.0GHz / 4GB RAM / Gigabit NIC

4 x Analog Network Encoder Servers
Description: These machines should be able to host 10 simultaneous encoding processes as well as 10 simultaneous client streams
Specs :
- Dual (Necessary?) Xeon @ 3.0GHz / 2GB RAM / Gigabit NIC
- 5 x Hauppauge PVR-500's (Link)
- 1 x Seagate 750GB SATA Drive (Link)
- 1 x AccelePort Xp 16-Port Multiport Serial Card PCI (Link)

Notes:
* Each output of the PVR500's are connected VIA S-Video to a DirecTV D10
* 10 of the 16 outputs of the AccelePort card are connected to the LSD (Low Speed Data) port of the D10s

2 x HD OTA Network Encoder Servers
Description: Similar in functionality to the Analog Network Encoder Servers except for HD content. These servers should be able to hold 5 simultaneous encoding processes as well as 5 simultaneous HD streams to clients.
- Dual (Necessary?) Xeon @ 3GHz / 2GB RAM (More?) / Gigabit NIC
- 5 Aver A180 OTA HD Encoders (Link)
- 3 x Seagate 750GB SATA Drives in RAID 5 (Link)

1 x Sat HD Network Encoder Server
- Single P4 3.2GHz / 1GB RAM (More?) / Gigabit NIC
- 3 x Seagate 750GB SATA Drives in RAID 5 (Link)

* This server will connect to 3 R5000s over 3 USB connections

Diagram:
(Attached Below)

Questions:

1) As I have understood the concept, the SageTV server will broker connections from Sage Clients directly to the servers that contain the media requested -- Is this the case?

2) Are the specs of my analog network encoders beefy enough to host 10 simultaneous (Analog) streams to clients as well as 10 encoding processes from the 10 encoders in the machine? Just looking at the specs, I would think not however I am hesitant to move to a 5 encoder per machine setup as this would double the amount of network encoders that I would have to run. Of course, that would also mean that each NE would only have to support 5 Encoding processes and 5 simultaneous streams to clients instead of 10 -- is this more reasonable?. Is it possible to instead use a striped array of 3 250GB disks (lower $/gb cost) to speed up the read/writes and then support a full 10 Encoding Processes & 10 Client Streams

Check My Math:
ENCODING; 3GB/HR Setting (on Sage) = 3072MB/HR = 51.2MB/Min = .85MB/s * 10 Encoders = 8.5MB/s Writing to Disk?

STREAMING; 3GB/HR Setting (on Sage) = 3072MB/HR = 51.2MB/Min = .85MB/s * 10 Encoders = 8.5MB/s Writing to Disk?

FULL LOAD (10 Encoding Processes / 10 Streams); 17MB/s Sustained Disk Activity, 8.5MB/s Network IO

3) Same question for my [u]HD Sat / OTA [u] network encoders

Check My Math:

FULL HD (Assuming full channel bandwith is used) = 19.34Mbs (Realistic for OTA?)
19.34Mbps = 2.4175 MB/s???? (Why does that not sound right????)

[digitalgm]

Sorry, I forgot to include the links to my post

Seagate 750GB HDD : Link

AccelePort Xp Multiport Serial Card : Link

AverMedia A180 : Link

NextCom R5000 : Link

--Regards, AdamR

[Opus4]

This is for a house... or a dorm?? Anyway, the one comment I have is:

Originally Posted by digitalgm

1) As I have understood the concept, the SageTV server will broker connections from Sage Clients directly to the servers that contain the media requested -- Is this the case?

I _think_ client connections would still have to go through the main server, not each network encoder, so the main server could be fairly busy. But, if the recording directories are specified using UNC paths, then the clients will be able to see the files on their own & play them directly w/o needing to stream the video through the server.

Also, I think you may need to look into defining which recording dirs the network encoders can use via the mmc/encoders/xxxx/forced_video_storage_path_prefix properties, so they don't all try recording to the same location.

- Andy

[dagar]

Wow! This takes hobby-shop to a new level

[Fluffdaddy]

Your Da-Man

[bcjenkins]

You may want to check out a new product from Magma which will let you add 6 pci slots onto 1 4x PCI-Express slot. Since all of the cards either have their own encoders or don't require encoders you might be able to get the umbers of servers down to one. Then you could start looking at quad boxes, or bigger. Since you seem to have a big system in mind and haven't mentioned a budget number, I would stay away from IDE and look at NAS, iScsi, blah blah blah.

Also, adapter teaming for NICs would help alleviate some network bottlenecks too.

[digitalgm]

Originally Posted by Opus4

This is for a house... or a dorm?? Anyway, the one comment I have is:

I _think_ client connections would still have to go through the main server, not each network encoder, so the main server could be fairly busy. But, if the recording directories are specified using UNC paths, then the clients will be able to see the files on their own & play them directly w/o needing to stream the video through the server.

Also, I think you may need to look into defining which recording dirs the network encoders can use via the mmc/encoders/xxxx/forced_video_storage_path_prefix properties, so they don't all try recording to the same location.

- Andy

Andy : Yes, believe it or not, this is for my home.
I should have made myself more clear in that I will be making each network encoder record to their local storage and use UNC paths so that the SageClients can directly access the media, hence the 750GB drive (or 3x250 striped array). I remember reading in another post (cant find it right now) that said if you specify UNC paths, the sage server will "forward" the connection to the network encoder to stream to you. However, this whole thing can be bust if the main sage server is going to have to stream all files because Im just not sure that is possible. Narflex, you there?????

Originally Posted by bcjenkins

You may want to check out a new product from Magma which will let you add 6 pci slots onto 1 4x PCI-Express slot. Since all of the cards either have their own encoders or don't require encoders you might be able to get the umbers of servers down to one. Then you could start looking at quad boxes, or bigger. Since you seem to have a big system in mind and haven't mentioned a budget number, I would stay away from IDE and look at NAS, iScsi, blah blah blah.

Also, adapter teaming for NICs would help alleviate some network bottlenecks too.

bcjenkins : I actually forgot to mention it, but I will be using a PCI enclosure from Magma - Their 7 slot one. (Link)

Actually, I HAVE to use an external enclosure since the servers I purchased for network encoders are 1U units and can only fit 1PCI card internally. Then, the plan was to use the Magma 7-Slot unit on each to house the actual encoder cards. I favor this solution because it still gives me a little room for expansion if I wanted (2 More PCI slots = 4 more encoders). Initially, I was very interested in the PCIe to PCI expansion but unfortunately my servers dont have PCIe slots, only PCI-X. Also, it seems like PCI-X solutions are going to be a lot more expensive and with the math that I have already done, I have calculated that I wont be hitting a bottleneck on the PCI bus even with all 10 encoders going at once so it may not justify the jump to PCI-X.

In reference to your IDE comment, I am staying FAAAAR away from it. The servers I am using have SATAII (300) connectors and I am planning to use that. Now, my initial thought was to use an enormous box as a file server (4Way Xeon, 8GB Ram, etc..) and let all of the network encoders send their files there but I read a thread that indicated this would not work for whatever reason. In fact, I even have some EMC equipment I was going to use for the job. Although, if we assume all 53 encoders are encoding at once to this box, theoretically we would be writing sustained ~40MB/s to the HDDs not including the HD broadcasts. Not sure how that is going to fare and if I want to invest a lot of money to find out

Anyways, thanks all of you for your comments but has anyone checked my initial math on disk writes? I really want to make sure those numbers are correct because that is what I am relying on for most of my decisions.

Attached : I took a picture of the server that I am using as my network encoder platform - it has the specs I listed in the Analog Network Encoder section. I currently own 5 of these units (4 Will be Analog Network Encoders and 1 will be the commercial skipping server as soon as I add 2GB of RAM to it.)

--Regards AdamR

[src666]

I just have to wonder about your tuner requirements. Are you really going to need to record 40 _different simultaneous_ SD shows? Likewise for 10 different simultaneous HD shows? I'm trying to imagine what possible need this could fill in a house, and I'm coming up blank.

As for your disc requirements, SATA probably won't cut it. You're going to need some hardcore throughput - Fiber Channel SCSI is probably your only choice. We're working with a dual format SAN at work (Fiber Channel SCSI and SATA), and the SATA is much, much slower than the SCSI.

I understand the "dream house" concept, but to me, your shopping list appears to need a reality check. Unless of course you really do need that many tuners - but I would _love_ to know why.

[Opus4]

Originally Posted by digitalgm

I should have made myself more clear in that I will be making each network encoder record to their local storage and use UNC paths so that the SageClients can directly access the media, hence the 750GB drive (or 3x250 striped array). I remember reading in another post (cant find it right now) that said if you specify UNC paths, the sage server will "forward" the connection to the network encoder to stream to you. However, this whole thing can be bust if the main sage server is going to have to stream all files because Im just not sure that is possible. Narflex, you there?????

Yes, that's what I said the purpose of the UNC paths is for -- so the client can see the file itself & play it directly rather than streamed via the server. I don't know that I would call it being "forwarded" to the network encoder to do the streaming; the client just reads it directly over the network w/o having another program be the streaming middle man.

The "I _think_" part was that all client connections have to go through the main server to get the list of recordings, etc. I shouldn't have used the term "think", because I'm sure that is the case -- there is only 1 overall server.

- Andy

[digitalgm]

Originally Posted by src666

I just have to wonder about your tuner requirements. Are you really going to need to record 40 _different simultaneous_ SD shows? Likewise for 10 different simultaneous HD shows? I'm trying to imagine what possible need this could fill in a house, and I'm coming up blank.

As for your disc requirements, SATA probably won't cut it. You're going to need some hardcore throughput - Fiber Channel SCSI is probably your only choice. We're working with a dual format SAN at work (Fiber Channel SCSI and SATA), and the SATA is much, much slower than the SCSI.

I understand the "dream house" concept, but to me, your shopping list appears to need a reality check. Unless of course you really do need that many tuners - but I would _love_ to know why.

scr666 : In reference to your question about my tuner requirements, I'll put it this way; lets say you have 3 TVs in your house. You would never want to run into a situation where somebody turns on the 3rd TV and is presented an error -- They cant watch TV because 2 people are already watching TV. So, just like that, where I have 20+ TVs in the home, I dont want to ever have an issue where I cant watch TV because someone else is watching. Its highly unlikely and near impossible that all 20+ televisions in my home will be on at once but I really want to plan for the worst. Also, as you already know in Sage, watching a television show live is the same as recording a show so where I may not have all 20 TVs on and watching shows, I may have things recording. I know it may not make sense to you, but its perfectly clear to me

As far as your disk thoughts, my first inclination was to use some EMC hardware I had lying around (CLARiiON) but after talking to some people it appears that I may be able to save some money and eliminate the need for more hardware by just allowing each network encoder to stream and store its own shows.

Opus4 : Perhaps my terminology is off but I think we are saying the same thing. By forwarded I only meant that the Client is first going to check the main SageTV server which will refer it to the location of the media file. Then, the client will stream the file from the UNC path of the network encoder that the media is located on. This is what you are saying, right?

[ToxMox]

I think you can do away with the Comskip dedicated server and instead run comskip on all the network encoder servers and run it live as the shows record. Those machines will be using hardly any cpu for just encoding since the cards will be doing all that work and that leaves plenty of CPU for running comskip on all the encoders for that machine.

[digitalgm]

Originally Posted by ToxMox

I think you can do away with the Comskip dedicated server and instead run comskip on all the network encoder servers and run it live as the shows record. Those machines will be using hardly any cpu for just encoding since the cards will be doing all that work and that leaves plenty of CPU for running comskip on all the encoders for that machine.

ToxMox : I know just the encoding iteslf will be barely using any CPU even if all 10 encoders on the machine are running at once. However, you have to assume that the machine is also serving 10 streams at once. If this is the case, then windows will be serving the streams over SMB (ugg) which is very disk and cpu intensive. Of course, I wont know until I try it. One of the first orders of business before getting this running is to actually simulate the environment with one Master SageTV server and one fully populated network encoding machines under the maximum amount of stress... We shall see

[ToxMox]

Some good points but I think you'll use an equal amount of CPU (or more) on the network encoder machine to have the Comskip server stream all the files to itself over the same SMB connection on the network encoder as it would to just run comskip locally on each network encoder.
You mentioned having to assume that the machine could potentially serve 10 streams at once but that number would actually be 20 (for all the clients) plus the 10 shows it could be recording to itself being streamed to the comskip server for a total of 30.

Also instead of going dual xeons you may want to consider using dual opteron (dual core) setups (save a little cash) giving each network encoder basically 4 CPUs to use which I would hope would be plenty to record 10 shows, run 10 simultaneous comskips as well as serve 20 streams.

[digitalgm]

Originally Posted by ToxMox

Some good points but I think you'll use an equal amount of CPU (or more) on the network encoder machine to have the Comskip server stream all the files to itself over the same SMB connection on the network encoder as it would to just run comskip locally on each network encoder.
You mentioned having to assume that the machine could potentially serve 10 streams at once but that number would actually be 20 (for all the clients) plus the 10 shows it could be recording to itself being streamed to the comskip server for a total of 30.

Also instead of going dual xeons you may want to consider using dual opteron (dual core) setups (save a little cash) giving each network encoder basically 4 CPUs to use which I would hope would be plenty to record 10 shows, run 10 simultaneous comskips as well as serve 20 streams.

Very good points ToxMox - if I did have a dedicated ComSkip server, I would have to host streams to the comskip server from the network encoders as well.... For some reason, I didnt consider that in this non-SAN iteration. I think its just goint to come down to actually setting up a 10 encoder server and trying to run all of this on it.

As far as your hardware thoughs, I agree with you that the Opertron server would be a MUCH more cost effective approach, however I already had some Dual Xeon 3GHz servers (4 Virtual CPUs like the opertron) that I wanted to put to use.

[Opus4]

Originally Posted by digitalgm

Opus4 : Perhaps my terminology is off but I think we are saying the same thing. By forwarded I only meant that the Client is first going to check the main SageTV server which will refer it to the location of the media file. Then, the client will stream the file from the UNC path of the network encoder that the media is located on. This is what you are saying, right?

Yes.

One other thing: That only applies to a regular SageTV Client. If you have any extenders (MVPs), those will always be going through the server... that is fine for mpeg2, since the decoding is done on the MVP. But, if you play a file type that needs to be transcoded, that will use the server's cpu. Unless I missed it, I'm not sure what type of clients you are planning to use.

- Andy

[digitalgm]

Originally Posted by Opus4

Yes.

One other thing: That only applies to a regular SageTV Client. If you have any extenders (MVPs), those will always be going through the server... that is fine for mpeg2, since the decoding is done on the MVP. But, if you play a file type that needs to be transcoded, that will use the server's cpu. Unless I missed it, I'm not sure what type of clients you are planning to use.

- Andy

Opus : My original thought was to have a mix of MVP/ME clients as well as wired HTPC's (SageTV Client). However, if what you are saying is true (and I have no reason to believe otherwise) I will probably change all of the MVP/ME clients to just full HTPC's.


Also, can someone please check my math :

ENCODING ANALOG; 3GB/HR Setting (on Sage) = 3072MB/HR = 51.2MB/Min = .85MB/s * 10 Encoders = 8.5MB/s Writing to Disk?

STREAMING ANALOG; 3GB/HR Setting (on Sage) = 3072MB/HR = 51.2MB/Min = .85MB/s * 10 Encoders = 8.5MB/s Reading From Disk?

FULL LOAD ANALOG; (10 Encoding Processes / 10 Streams); 17MB/s Sustained RW Disk Activity, 8.5MB/s Network IO

FULL HD (Assuming full channel bandwith is used) = 19.34Mbs (Realistic for OTA?)
19.34Mbps = 2.4175 MB/s???? (Why does that not sound right????)

AdamR

[GKusnick]

Originally Posted by digitalgm

I will probably change all of the MVP/ME clients to just full HTPC's.

So now you're up to, what, around 30 computers dedicated to this system? You're going to need a full-time IT manager just to keep the thing running. Rolling out a new release of Sage to all those machines will likely involve a day or two of downtime -- never mind any customizations or add-ons.

So if the goal of all this excess capacity is to ensure that you never see an error screen, I don't think you've achieved that goal. There's still going to be maintenance downtime, and the bigger and more complex the system is, the longer it's going to take to carry out those maintenance chores. And in the end I suspect you're going to find that you've got an awful lot of hardware sitting idle 99% of the time.

[Deadbolt]

Originally Posted by GKusnick

So now you're up to, what, around 30 computers dedicated to this system? You're going to need a full-time IT manager just to keep the thing running. Rolling out a new release of Sage to all those machines will likely involve a day or two of downtime -- never mind any customizations or add-ons.

So if the goal of all this excess capacity is to ensure that you never see an error screen, I don't think you've achieved that goal. There's still going to be maintenance downtime, and the bigger and more complex the system is, the longer it's going to take to carry out those maintenance chores. And in the end I suspect you're going to find that you've got an awful lot of hardware sitting idle 99% of the time.

Shhhhhhhh! I wanna see pictures!

[digitalgm]

Originally Posted by GKusnick

So now you're up to, what, around 30 computers dedicated to this system? You're going to need a full-time IT manager just to keep the thing running. Rolling out a new release of Sage to all those machines will likely involve a day or two of downtime -- never mind any customizations or add-ons.

So if the goal of all this excess capacity is to ensure that you never see an error screen, I don't think you've achieved that goal. There's still going to be maintenance downtime, and the bigger and more complex the system is, the longer it's going to take to carry out those maintenance chores. And in the end I suspect you're going to find that you've got an awful lot of hardware sitting idle 99% of the time.

Greg : For the last 2 or so years I have had Sage running perfectly stable. I attribute this to two factors - the first of which being that I stayed on the same version and never upgraded or "tried out" new STVs or STVi's. In addition, I had it running for a while using a non-commit file system so everytime the system booted up, it was just as it was installed the first time. I can honstly say I didnt have any problems with this type of a setup.

Now, when a new version came out that was advertising some more features, (and this happened more than once while I had Sage) I simply added it to my QA box which was setup in my office and tested it out for a couple days. I usually keep the TV in my office on and use it throughout the day so I could pretty simply identify when problems would arise. Once I would deem the version as stable I would create an image of it and apply it to the clients in my house. I will admit that it took a while for me to get a setup that I really liked but once I got there I kept it.

With this new setup, I will be using GhostCast to broadcast the images to the HTPC Clients. Basically all I am going to have to do is create a reference image like I did before and then apply the image to all of the clients whenever I feel a new version is necessary. Simple Also, to ensure that when I do roll out new images all goes well, all of the clients will be identical.

As to whether or not the hardware will be idle; it may very well be. In fact I will most likely begin this project with only 1/3 of what I have specced out to see how it performs and expand as necessary.

@ Deadbolt : I promise pictures - In fact, I think I posted pics of the servers that will be functioning as analog network encoders. I'll take more as soon as the house actually exists Its still under construction.

[teknubic]

Originally Posted by Deadbolt

Shhhhhhhh! I wanna see pictures!

Haha
I suspect an imaging process would be ideal in his case but good point nonetheless.

I'm curious about how you're going to overcome the heat generated from all those procs and tuners. I have a PVR500 and a pair of vBox 151s in only 1 computer and it heats the office nicely.

Also, if we're in the same area, I'd love to offer my services in any capacity I can, Adam.