External USB drive and no. of streams

[Sparhawk6]

I think I have seen this info around before, but I can't find it. How many streams could I record over USB 2.0 to an external drive?

How many HD streams? How many SD streams? In other words, when would I hit the 480Mbps cap?

[GKusnick]

Do the math. SD runs about 2-3 GB/hr, depending on quality. HD is about 6-8 GB/hr. 480 Mbps = 216 GB/hr. So even figuring only about half of that as useful bandwidth, that's still enough for upwards of 12 HD streams or 30 SD streams.

Of course the performance of the drive itself may be less than that. But even cutting it in half again, there's still a lot of headroom for multiple streams.

[stevech]

For quite a while, my Sage server was a 1.6GHz laptop with 3 USB2 drives on one hub. And an 80GB partition on the laptop's 2.5in. drive. All formatted at 64KB block sizes. The large block size helps reduce the I/O calls per second a LOT. With one tuner, also USB2, on a separate hub, I never had hosed up recordings.

my measurements show that one standard def stream is about all you should try on a USB2 drive. Not sure it could do HD at all, reliably, nor record and playback on same drive.

Also this would apply to the capacity of a given hub (non-daisy-chained), so a second stream on the same hub may not work well.

I see a dramatic difference in speed of 64K block formatted partitions on USB2 vs. PCI and SATA-II.

ALso, CPU speed and loading seems to affect USB2 speed quite a bit.

No doubt, someone will disagree.

[Sparhawk6]

Wow, so one person says only 1 SD stream, while another says 12 HD streams! Quite the difference! Can anyone confirm with actual experience?

[dgbruce]

As a practical example, I regularly record 4 SD streams and 1 HD stream , and play back a stream (HD or SD) from a JBOD array (4 X 500G WD HD on a single USB interface) without any issues... no stuttering, no recording errors...

[GKusnick]

As I write, I have two HD streams playing back from the same USB drive without issues. (I'm using WMP for this test, because the drive in question isn't hooked up to my Sage machine at the moment.) Perfmon shows disk activity in the 3 MB/s range. A few minutes ago I ran a throughput benchmark on this drive and got figures in the range of 20 MB/s. So that seems consistent with the idea of 10-12 simultaneous HD streams.

Again, a lot will depend on what's at the end of the USB bus. The bus itself will not likely be the bottleneck. But with a reasonably fast drive, I don't see why you shouldn't be able to support several simultaneous streams.

[sainswor99]

Originally Posted by GKusnick

As I write, I have two HD streams playing back from the same USB drive without issues. (I'm using WMP for this test, because the drive in question isn't hooked up to my Sage machine at the moment.) Perfmon shows disk activity in the 3 MB/s range. A few minutes ago I ran a throughput benchmark on this drive and got figures in the range of 20 MB/s. So that seems consistent with the idea of 10-12 simultaneous HD streams.

Again, a lot will depend on what's at the end of the USB bus. The bus itself will not likely be the bottleneck. But with a reasonably fast drive, I don't see why you shouldn't be able to support several simultaneous streams.

It's not just the speed of the drive; it's also the CPU. USB is CPU dependent. I have no idea what the relationship is between the CPU and USB; I just know that one exists. You should be able to get more than one stream going at a time, however, assuming that you have a relatively powerful machine.

How many licks does it take to get to the center of a Tootsie Roll pop? The world may never know.

[sandor]

theoretical maximums vs. actual use


i've ditched all the usb drives i had for internal SATA - i really couldn't get more than 2 HD streams (i don't view any SD) on the USB drives. same drives on firewire were able to handle 3+ with no problems, SATA the same.

i still try to have at least one unique drive for each of my tuners (i have 3 tuners, and 4 drives right now).

[GKusnick]

Originally Posted by sainswor99

It's not just the speed of the drive; it's also the CPU.

Sure; there are lots of factors that can affect overall throughput. But the original question (at least as I read it) was about bus bandwidth:

when would I hit the 480Mbps cap?

I.e. how many streams would it take to saturate the bus? Obviously if your system is sub-par in other areas, you'll never get to that point, but that's the question I was trying to answer.

[sainswor99]

Originally Posted by GKusnick

Sure; there are lots of factors that can affect overall throughput. But the original question (at least as I read it) was about bus bandwidth:


I.e. how many streams would it take to saturate the bus? Obviously if your system is sub-par in other areas, you'll never get to that point, but that's the question I was trying to answer.

Sorry, I should have also clarified; USB is affected by CPU more than firewire, for example. Again, I'm not sure what the relationship is, but I do know that if you try to write something to a USB drive vs writing to a firewire attached drive, the usage is dramatically different. Unless you have a incredibly powerful CPU, you'll overwhelm it long before you saturate the bus. So attempting to extrapolate the number of potential streams is difficult to do for USB without knowing what the CPU is; you can come up with a better estimate for firewire without knowing that information.

[Sparhawk6]

Originally Posted by sainswor99

Sorry, I should have also clarified; USB is affected by CPU more than firewire, for example. Again, I'm not sure what the relationship is, but I do know that if you try to write something to a USB drive vs writing to a firewire attached drive, the usage is dramatically different. Unless you have a incredibly powerful CPU, you'll overwhelm it long before you saturate the bus. So attempting to extrapolate the number of potential streams is difficult to do for USB without knowing what the CPU is; you can come up with a better estimate for firewire without knowing that information.

Good to know. I have an athlon XP 3800. I wonder how many streams that could handle...

[stevech]

Originally Posted by Sparhawk6

Good to know. I have an athlon XP 3800. I wonder how many streams that could handle...

I too run an AMD X2 (dual core) 3800 for Sage as a server (service mode). I measured a very large improvement in speed going to SATA 2 (RAID1) from USB2, and a large decrease in CPU usage.

This AMD X2 server has super low CPU usage, running Sage (no HD) plus two other 24/7 applications related to home automation and web browsing via an S-Video link to the TV set.

The AMD X2 3800 was chosen due to low power consumption, runs very cool, and is low cost and uses an ASUS microATX motherboard - same one I have in an office PC that's an AMD X2 4200. Very reliable.

I think USB2 uses a lot of CPU to do data moves related to packetizing USB2 data streams, and vice-versa. I recall that USB2 in this mode also has a CRC check for the CPU to do. In SATA and Ethernet, this is done in the controllers.

Isn't HD with mpeg2 up to 12Mbps per stream? Perhaps it's related to interlaced or not, but Standard Def seems to be about one fourth that rate.

[Menehune]

Originally Posted by sainswor99

It's not just the speed of the drive; it's also the CPU. USB is CPU dependent. I have no idea what the relationship is between the CPU and USB; I just know that one exists. You should be able to get more than one stream going at a time, however, assuming that you have a relatively powerful machine.

How many licks does it take to get to the center of a Tootsie Roll pop? The world may never know.

Firewire is claimed to have 1/2 the CPU utilization of USB. I have seen benchmarks of combo USB/FW external drives that show FW CPU utilization figures of 10% and USB CPU utilization of 15% during benchmark testing.

Since HD recording is very processor intensive (I think I read on these boards around 40% CPU utilization per stream?)-that could mean that if you have an external USB drive and you are recording HD, you may run into CPU bottlenecks if you try to record more than 2 HD streams to an external USB drive. Throw in transcoding to a MVP and you may run into glitching.

Personally, I avoid USB if I am going to copy any large files (greater than 200MB) since my computers always seem to default to USB1.1 speeds and the only way I can get any acceptable file transfer times is to use firewire HDs.

[GKusnick]

Originally Posted by sainswor99

Unless you have a incredibly powerful CPU, you'll overwhelm it long before you saturate the bus.

I've moved my USB drive to my SageTV server and added it as an imported video directory. I am right now playing back two HD streams off that drive through two separate machines running SageTVClient. (No playback on the server itself.) CPU usage is hovering around 5%, disk I/O around 3 MB/s. Based on those numbers, I don't see a problem with scaling to, say, 10 streams @ 25% CPU and 15 MB/s (1/4 the nominal USB2 bandwidth).

This is with an Athlon 64 X2 3800+ 2.0 GHz, not a super-high-end CPU when I bought it last year, even less so today.

Originally Posted by Menehune

Since HD recording is very processor intensive (I think I read on these boards around 40% CPU utilization per stream?)...

OK, I've just stopped playback and switched to recording. I've got two simultaneous HD streams recording to my SATA drive (not USB). CPU usage is in the neighborhood of 6-7%, disk I/O around 4 MB/s. So HD recording per se is not a CPU hog (nor would I expect it to be, since all it's really doing is writing the incoming digital stream to disk).

Putting it together, I figure a total of around 6% CPU per HD stream recording to USB. (I didn't actually do this experiment; I'm just adding up the numbers from the previous two experiments.) That scales to 60% CPU for 10 simultaneous streams.

Bottom line is that in these real-world tests I'm just not seeing the high CPU demand that some posters are predicting, nor any other obvious barriers to recording several HD streams at once to USB.

[Opus4]

USB2 implementations can vary quite a bit -- people who have had problems with even 1 Hauppauge PVR-USB2 had no problems after switching to a different USB2 card, for example. Anyone who has very poor USB2 performance should consider buying a new USB2 card.

- Andy

[sainswor99]

Originally Posted by GKusnick

I've moved my USB drive to my SageTV server and added it as an imported video directory. I am right now playing back two HD streams off that drive through two separate machines running SageTVClient. (No playback on the server itself.) CPU usage is hovering around 5%, disk I/O around 3 MB/s. Based on those numbers, I don't see a problem with scaling to, say, 10 streams @ 25% CPU and 15 MB/s (1/4 the nominal USB2 bandwidth).

This is with an Athlon 64 X2 3800+ 2.0 GHz, not a super-high-end CPU when I bought it last year, even less so today.


OK, I've just stopped playback and switched to recording. I've got two simultaneous HD streams recording to my SATA drive (not USB). CPU usage is in the neighborhood of 6-7%, disk I/O around 4 MB/s. So HD recording per se is not a CPU hog (nor would I expect it to be, since all it's really doing is writing the incoming digital stream to disk).

Putting it together, I figure a total of around 6% CPU per HD stream recording to USB. (I didn't actually do this experiment; I'm just adding up the numbers from the previous two experiments.) That scales to 60% CPU for 10 simultaneous streams.

Bottom line is that in these real-world tests I'm just not seeing the high CPU demand that some posters are predicting, nor any other obvious barriers to recording several HD streams at once to USB.

Not that I really want to get into a geek fight about this (mainly because I don't know enough to really represent my case well), but you didn't exactly test what I suggested. I was suggesting that CPU utilization would be high when WRITING to a USB disk, especially compared to other interface options. Your tests were based on reading from a USB interface and writing to a SATA interface.

You may still be correct; what little research I did indicated that as USB standards have matured, this CPU utilization has improved with each iteration. In other words 2 > 1.1 > 1; 2.1 (has it been released yet?) is supposed to be equivelent to firewire.

[GKusnick]

Originally Posted by sainswor99

...you didn't exactly test what I suggested.

Yes, I was aware of that, but there's a limit to how much reconfiguration of my Sage server I'm willing to do for this.

However if someone else wants to do the experiment and post the results, I'd be interested to see them.

I'm not really looking to get into a fight here either. It just seemed to me that the thread could use a bit more hard data to counterbalance some of the guesswork.

[sandor]

Originally Posted by Menehune

Since HD recording is very processor intensive (I think I read on these boards around 40% CPU utilization per stream?)

recording HD is NOT processor intensive at all. There is nothing involved in recording HD but writing an MPEG stream to disk. At the moment i am recording 2 HD (ATSC) streams and using 2 % CPU. HD is fairly disk intensive, as each stream will be needing a steady 2.3 MB/s or so (max 19 mbps) - so while 3 streams is less than 9 MB/s, it is the concurrent writes that ATA has always had more of a problem with than say SCSI - thus the reason native command queueing is such a big deal on SATA (hurrah!)

[Menehune]

Originally Posted by sandor

recording HD is NOT processor intensive at all. There is nothing involved in recording HD but writing an MPEG stream to disk. At the moment i am recording 2 HD (ATSC) streams and using 2 % CPU. HD is fairly disk intensive, as each stream will be needing a steady 2.3 MB/s or so (max 19 mbps) - so while 3 streams is less than 9 MB/s, it is the concurrent writes that ATA has always had more of a problem with than say SCSI - thus the reason native command queueing is such a big deal on SATA (hurrah!)

I don't have HD so unfortunately I'm relying upon my recollections of threads. I didn't think just recording an already encoded bit stream would take that much cpu since it has already been converted at the head end. I must have been thinking about transcoding a program to an MVP then as being very processor intensive. I must've been HD playback that required horsepower as well.

-back to lurking.

[sandor]

Originally Posted by Menehune

I don't have HD so unfortunately I'm relying upon my recollections of threads. I didn't think just recording an already encoded bit stream would take that much cpu since it has already been converted at the head end. I must have been thinking about transcoding a program to an MVP then as being very processor intensive. I must've been HD playback that required horsepower as well.

-back to lurking.

yeah, transcoding HD is kind of insane CPU-wise, it is just a lot of bits to process. and your instinct on recording HD is/was spot on!