JBOD Question

[bigo93bronco]

My server currently has (4) 750GB drives, and (1) 1TB drive configured as just individual drives to store all of my saved media (not the recording drive). The problem is that the drives have alot of hours on them now, they waste alot of space, and i'm running out of space. I am interested in creating a new JBOD array using a quality hardware controller and 3-5 2TB sata drives. Once I have everything on the new drives, I will retire the old drives to a safe place and use them as a permanent backup. I know JBOD doesn't offer fault tollerance, but is there any way I can set it up so that a single drive failure only results in data lost on the faulty drive? If I can get this, I will be able to just replace the bad drive, and then reload it from my backup drives. This would be much faster than having to reload the entire JBOD array. Is there any card or manufacturer that supports something like this?

I have been looking all around the internet for JBOD and Spanning information, and it seems that every site, author, and manuafacturer contradicts the other. I am familar with the traditional RAID sets, but I have never really used JBOD before.

[Nelbert]

I suspect it will depend on the implementation of JBOD that is used to present the drives as a single volume and that you'll need a software based solution rather than a hardware based one.

Striped JBOD will lost everything as it's basically no different to RAID 0, linear JBOD potentially you only lose that drives worth as each drive is added to the end of the next one, but depending on the implementation you could still lose everything, especially if it's handled in the controller.

If the controller presents all the drives to the OS as a single large drive, there is no way the controller can recover the lost drive data. The OS thinks it has a single drive so uses all the disks as a single large disk. It has no control over where data is stored and no way of telling that there are actually 5 disks not 1. When a disk fails what have you lost? media files? system files? To the OS a large part of the single disk has suddenly failed.

With a software based solution you have a chance that it may work. eg under linux LVM in linear mode, metadata records on every disk list which blocks are on each physical disk on every drive in the array, and the drives are filled up sequentially, so when a drive fails potentially you just lose that part of the volume and the other drives are readable. In the MacOS implementation the data on the remaining drives is useless when a single disk in an JBOD array fails.
With Windows, Drive Extender was the solution, until it was canned.

Could you not just have a small mirrored pair (eg 2 x 160Gb) for the OS and then just mount the new drives as folders rather than drive letters? All the drives could then appear under a single media folder, eg movies_1 photo_1 movie_2 photo_2, if a drive went you'd only lose the data on that drive. It would still be a manual management of what to store on which drive though unlike a JBOD raid implementation.

[davephan]

You might consider using unRAID if you have a second computer system that will boot from a flash drive. UnRAID has redundancy. You lose nothing with one drive failure. You lose only one drive if you have two drive failures. You can use different drive sizes, except the parity drive has to be larger or equal to the largest of your drives. The unRAID server could be used for your video library and the smaller drives for the recordings.

Before you buy the new 2 TB drives, you should read the reviews on the drives. Some people have a lot of trouble with the cheaper 2 TB green drives. All my 2 TB are the cheaper green drives, and I haven't any significant trouble yet. I don't pack the drives with too much density in the case, which allows better ventillation that might increase the drive life.

Dave

[bigo93bronco]

My biggest issue is that I am wasting a bunch of space on each individual drive. I have way too much media to use any kind of mirroring RAID. I'd just be happy to have some kind of JBOD or spanning solution where I can expand the array without having to wipe and reformat it, and where a single drive failure only results in the loss of data on that particular drive. I tried using windows integrated spanning once on my vista system, and couldn't get it to work. I may just have to go with some kind of hot-swap raid 5 setup. These drives have nothing to do with tv playback or recording, so it shouldn't cause any issues right?

[davephan]

Originally Posted by bigo93bronco

My biggest issue is that I am wasting a bunch of space on each individual drive. I have way too much media to use any kind of mirroring RAID. I'd just be happy to have some kind of JBOD or spanning solution where I can expand the array without having to wipe and reformat it, and where a single drive failure only results in the loss of data on that particular drive. I tried using windows integrated spanning once on my vista system, and couldn't get it to work. I may just have to go with some kind of hot-swap raid 5 setup. These drives have nothing to do with tv playback or recording, so it shouldn't cause any issues right?

UnRAID might be your solution if you have an extra computer that can boot from a flash drive. You can use a variety of different sized drives. You'd need to start with a parity drive and a data drive. The parity drive needs to be the largest one in the system and can be upgraded to an even larger one later. UnRAID will protect your files from the loss of a single drive. After setting up the unRAID system, you would transfer over one disk at a time, then add the disk to the unRAID system after you transfer all the data. You would add one drive at a time like this until all your drives are in the unRAID system. The downside to unRAID is the transfer throughput, but the throughput is fast enough for several HD streams.

Dave

[bigo93bronco]

That sounds like exactly what i'm looking for. So is the total storage on unraid N-1 like with raid 5? Do you have to run the array on a seperate computer? why?

[davephan]

Originally Posted by bigo93bronco

That sounds like exactly what i'm looking for. So is the total storage on unraid N-1 like with raid 5? Do you have to run the array on a seperate computer? why?

There are three versions of UnRAID. Basic 3 data drives, Plus 6 data drives, Pro 20 data drives. The Basic version is totally free, so you can try it out for free. If you like unRAID, you will probably want to purchase the Plus or Pro version. I bought two pre-configured USB flash drives with the Pro version.

http://www.lime-technology.com/produ...istration-keys

UnRAID is run on a separate computer from your SageTV computer. The hard drives are simply connected to your on-board drive controller and any additional add-on drive controllers. No operating system / boot drive is needed for unRAID. The unRAID server boots from the USB flash drive. The unRAID computer has to be new enough so that the system board can boot from a USB flash drive. The operating system used for unRAID is a form of Linux. Updates are released from Lime Technologies infrequently. No patching is needed. The unRAID server simply runs for long periods without any need for rebooting. Some people on the forum were trying to get unRAID and SageTV to run on the same system using VMs. I don't know if anyone has been successful yet.

I setup a separate system for my unRAID server. I have 16 TB of storage in my unRAID server and almost 6 TB in my SageTV server. The recordings are done by the SageTV server. I periodically delete files from the SageTV server or move them to the unRAID server if I want to keep them. You can grow the unRAID server pretty large, up to about 36 TB with all 2 TB drives. Even higher if you used 3 TB drives.

The total storage is similar to RAID 5 but unRAID is supposed to be closest to RAID 4. There is one parity drive. The parity drive must be equal to or larger than all of your data drives. The Plus and Pro versions have a 'cache' drive, which can speed up the data transfer.

Unlike RAID, the drives can be different types, sizes, and speeds. The total storage available is the sum of all your data drives. This is far more efficient storage than WHS where only 50% of your data drive storage is available.


Dave

[Fuzzy]

Back to the original question though, what are you trying to accomplish by using JBOD? I ask because you mentioned no desire for fault tolerance, and only wish to lose a single drives worth of data on a drive failure - that would lead me to state that keeping the drives separate is the best option. You will have far better performance in sage with 4 individual drives, than you will with 4 drives in a linear JBOD. Multiple drives versus a single drive is also completely transparent to sage.

[bigo93bronco]

I have a huge collection of tv shows organized by season. I am at the point now, where I have a GB or so left on each disk, but not enough room to put anything new on the server. If I had all the drives in some type of array, all that space would add up to something I could actualy use. I just would like something that handle one drive failure without reloading the entire 4TB+ array. I like the unraid, but I want to keep everything in one machine. I also want something that can be expanded to hold more disks without requiring a complete rebuild (tall order I know).

[Fuzzy]

That's kind of my point. You don't really NEED to have them all in a single volume with sagetv. Even if you DO have them in folders structures and such (though I feel there are better ways of storing them than this), sage will combine multiple import paths into a single folder display.

That said, You might look into FlexRAID - not necessarily for the RAID style protection, but for FlexRAID-View, which is a way of combining multiple file systems and presenting them as a single virtual volume space.

[rrhorer]

Originally Posted by bigo93bronco

If I had all the drives in some type of array, all that space would add up to something I could actualy use.

First, from everything you've said, I believe Fuzzy is absolutely right (post #8) about using SageTV and individual drives. And, I doubt that you will squeeze out that much more useful storage with a different solution.

I just would like something that handle one drive failure without reloading the entire 4TB+ array. ... I want to keep everything in one machine. I also want something that can be expanded to hold more disks without requiring a complete rebuild (tall order I know).

While there may be a good raid solution for one or more of your "wants", sage probably offers the easiest and best solution for the combination of needs you have indicated.

[bigo93bronco]

So basicaly what your saying, is to just keep the drives seperate, and store media anywhere it will fit rather than TV/showname/season/episodes, and let sagetv organize it into one library. Thats what I am allready doing right now (except I use the folders structure) using the "folders" view in the video library. I would probably have to rename alot of my media in order to do this, or use alot of mappings. I'll take a look at the "flex-raid" next, and see if it will work for me. I don't want to add anything outside of server, because it has a triple redundant power supply, and the entire system can run for thirty minutes off of the UPS I have it plugged into. I was actually hoping to reduce the number of phisical disks while increasing the capacity at the same time.

[Fuzzy]

You can still keep them in TV/Showname/Season/episode format - but you can spread them between drives, and it will be transparent:

M:\TV\Lost\Season 01\1 - Pilot - Part 1
M:\TV\Lost\Season 01\2 - Pilot - Part 2
M:\TV\Lost\Season 01\3 - Tabula Rasa
N:\TV\Lost\Season 01\4 - Walkabout
N:\TV\Lost\Season 01\5 - White Rabbit
O:\TV\Lost Season 01\6 - House of the Rising Sun

will all display together in sage under Lost\Season 01 if you have M:\TV; N:\TV and O:\TV as your import folders.

[bigo93bronco]

lol.....those are allready part of my collection. The only reason I am hesitant to do that, is that it makes it harder to back things up, or transfer to other devices (with it spread out). Sounds like I have some thinking to do. I'm not quite ready to do this anyway, and I may hold off a bit for 3TB drives to become affordable. I see my final storage requirements being 15-16TB someday, and I'm doing my best to find a solution that can be expanded reliably over time to reach that mark. I appreciate the suggestions and patience from everyone.

[Fuzzy]

Well, that's also why we were recommending separate drives. They are without a doubt the easiest expansion method, as you simply... add a new drive. no worrying about extending arrays or anything. And if you want to replace a 1TB, say, with a 3TB, you can simply copy the partition over to the new drive, extend it (easy with GPT partitioned NTFS drives in windows), and add more to it.

[Fuzzy]

As mentioned, the ONLY reason when dealing with sage, to use something like UnRAID (or any raid solution) is for fault tolerance.

[bjterry62]

Originally Posted by bigo93bronco

lol.....those are allready part of my collection. The only reason I am hesitant to do that, is that it makes it harder to back things up, or transfer to other devices (with it spread out). Sounds like I have some thinking to do. I'm not quite ready to do this anyway, and I may hold off a bit for 3TB drives to become affordable. I see my final storage requirements being 15-16TB someday, and I'm doing my best to find a solution that can be expanded reliably over time to reach that mark. I appreciate the suggestions and patience from everyone.

Thought I'd add my 2 cents.

I'm with davephan, I planned to just use single drives and folders when I started with Sage, but I quickly outgrew the storage space I had. I built an unRAID basic server out of an old beige case, an MSI RS480M-IL socket 754 MB, mobile AMD processor, 512mb ram, Intel gigabit nic ( from ebay. unRAID didn't like the on-board nic) and 3 750gb SATA drives. Literally stuff I had laying around. This setup lasted me a little over 2 years till I recently upgraded the case and added some more drives. Best part is, if you have junk parts laying around, you can test to see if unRAID will work on it for nothing. Visit the unRAID site and read up on it a bit. It's the simplest server setup I have found.

BT

[Fuzzy]

UnRAID would fit the bill for media storage - but for recording, it rapidly becomes network limited AND drive speed limited. Plus, it's an extra system up and running 24/7. Putting sagetv recordings over the network are VERY taxing. Especially when you start looking at multiple live watching, from network connected tuners. For instance, running two ATSC streams from an HDHR to the sagetvserver - then to the UnRAID for recording - then back to the server for playback - then from the server to the extenders. This, including the overhead, will pretty much max out a gig-e network connection. Some of it can be alleviated with multiple NIC's on the server (one for the house network/extenders, and another dedicated link to the unraid machine - and another dedicated nic for the HDHR's... again... a lot of extra complexity, and with that extra complexity, you have to decide what you are GAINING using the unraid machine.

[rrhorer]

Originally Posted by bjterry62

Thought I'd add my 2 cents.

That's what I did; and I still believe that Fuzzy has consistently offered good advise for what has been asked. So, at this point, I want my 2 cents back. Just kidding; but you either need to define what you need and not what you want or accept the fact that what you want may be something different than what you really need.

[stevech]

Originally Posted by bigo93bronco

My server currently has (4) 750GB drives, and (1) 1TB drive configured as just individual drives to store all of my saved media (not the recording drive). The problem is that the drives have alot of hours on them now, they waste alot of space, and i'm running out of space. I am interested in creating a new JBOD array using a quality hardware controller and 3-5 2TB sata drives.....

I simply use pairs of 500GB (could be larger now, with the lower prices), in RAID1 (mirror done in the motherboard chip). One drive fails, I lose no data. Drives are cheap. JBOD is too risky for me.