• Digital Bolex available to order
  • Kish/Bolex Series 1 Primes on sale 9/15
  • D16M Monochrome: The world's most affordable black and white cinema camera
  • F&V Products in stock in the Digital Bolex Store

Please consider registering
guest

Log In Register

Register | Lost password?
Advanced Search

— Forum Scope —

  

— Match —

   

— Forum Options —

   

Minimum search word length is 4 characters – maximum search word length is 84 characters

Topic RSS
RAID solutions for D16 footage
December 10, 2012
9:18 am
James M
NTX
Member
Forum Posts: 606
Member Since:
March 28, 2012
Offline

I’m currently upgrading the PC tower I do my editing on and starting thinking ahead to working with Cinema DNG files. And while I’m a big fan of CalDigit and G-Technology I’m thinking I’d like to build my own RAID enclosure. 

Joe, you mentioned “the write speed is 145MB/s we estimate that at 2k 24fps the D16 will require 84MB/s.” So would a RAID 5 or RAID 6 solution with a similar sustained read/write speed do the trick? I’m even considering some USB 3.0 solutions since my laptop has the ports built in and I’m adding a USB 3.0 PCIE card to my tower.

Just curious if anyone has any previous or current DIY experiences they could share.

December 10, 2012
10:16 am
joerubinstein

Camera Owner
Forum Posts: 1789
Member Since:
February 28, 2012
Offline

Hey James,

 

Yeah a custom built Raid 5 or 6 would work well, you would need a separate back up drive, but USB 3 drives are getting to be pretty cheap. I built a 6TB raid inside of a PC tower recently, but not in a dedicated enclosure so I don’t know which to recommend. USB 3 is definitely fast enough though.

 

Let us know what you come up with!   Joe

January 7, 2013
8:33 am
IndySkint
Dublin & Orlando

Camera Owner

Kickstarter Backer
Forum Posts: 117
Member Since:
April 19, 2012
Offline

I have to say that my biggest headache at the moment is data storage. Raid is the only viable solution I see, especially given the fact that I’ll be shooting a feature on the D16. I came cross LaCie recently and I like their approach – 

 

http://www.lacie.com/ie/produc…..m?id=10061

 

Could definitely be a option. Mac and PC friendly, but different flavors for each.

January 7, 2013
2:24 pm
joerubinstein

Camera Owner
Forum Posts: 1789
Member Since:
February 28, 2012
Offline

That Lacie raid seems awesome!

 

And now 4TB internal drives are only $259 I think we just halved the hard drive price again :)

March 3, 2013
1:07 pm
Jade Lane

Camera Owner

Kickstarter Backer
Forum Posts: 156
Member Since:
April 23, 2012
Offline

James M said
I’m currently upgrading the PC tower I do my editing on and starting thinking ahead to working with Cinema DNG files. And while I’m a big fan of CalDigit and G-Technology I’m thinking I’d like to build my own RAID enclosure. 

Joe, you mentioned “the write speed is 145MB/s we estimate that at 2k 24fps the D16 will require 84MB/s.” So would a RAID 5 or RAID 6 solution with a similar sustained read/write speed do the trick? I’m even considering some USB 3.0 solutions since my laptop has the ports built in and I’m adding a USB 3.0 PCIE card to my tower.

Just curious if anyone has any previous or current DIY experiences they could share.

I’d like to pick this topic up again as its causing me some headache and confusion too.

 

I was just about to buy a G-Tech 8TB G-Raid Thunderbolt HD that in the spec states Raid 0, you mention needing a Raid 5 or 6 for the D16 CinemaDNG files and footage and also mentioned the Lacie Raid drives but their spec is not Raid 5 or 6 (unless I’m not looking properly or don’t understand).

 

Can those that actually know or understand the RAID storage situation make a list of suitable drives (USB 3 & Thunderbolt) that will be a correct solution for handling the D16 footage and at what kind of length footage over TB space storage etc.

 

March 3, 2013
1:21 pm
Ari Davidson

Camera Owner

Kickstarter Backer
Forum Posts: 379
Member Since:
March 17, 2012
Offline

Jade Lane said

James M said
I’m currently upgrading the PC tower I do my editing on and starting thinking ahead to working with Cinema DNG files. And while I’m a big fan of CalDigit and G-Technology I’m thinking I’d like to build my own RAID enclosure. 

Joe, you mentioned “the write speed is 145MB/s we estimate that at 2k 24fps the D16 will require 84MB/s.” So would a RAID 5 or RAID 6 solution with a similar sustained read/write speed do the trick? I’m even considering some USB 3.0 solutions since my laptop has the ports built in and I’m adding a USB 3.0 PCIE card to my tower.

Just curious if anyone has any previous or current DIY experiences they could share.

I’d like to pick this topic up again as its causing me some headache and confusion too.

 

I was just about to buy a G-Tech 8TB G-Raid Thunderbolt HD that in the spec states Raid 0, you mention needing a Raid 5 or 6 for the D16 CinemaDNG files and footage and also mentioned the Lacie Raid drives but their spec is not Raid 5 or 6 (unless I’m not looking properly or don’t understand).

 

Can those that actually know or understand the RAID storage situation make a list of suitable drives (USB 3 & Thunderbolt) that will be a correct solution for handling the D16 footage and at what kind of length footage over TB space storage etc.

 

I believe Raid configurations are done on the user’s end. You can use disk utility to do it (http://macperformanceguide.com…..pRAID.html)

Here’s some info on what a RAID entails: http://macperformanceguide.com…..pRAID.html

Here’s a visual representation of what your drives are doing Laugh Watercooler_63190a_1433997.jpg

 

Basically you need at least three drives for the data to be striped and allow for a failure of one drive without data loss or corruption. 

March 3, 2013
1:23 pm
Ari Davidson

Camera Owner

Kickstarter Backer
Forum Posts: 379
Member Since:
March 17, 2012
Offline

Also there’s a company called drobo (http://www.drobo.com/products/…../index.php) which supposedly makes the whole process as easy as plug and play. 

March 3, 2013
2:18 pm
Jade Lane

Camera Owner

Kickstarter Backer
Forum Posts: 156
Member Since:
April 23, 2012
Offline

Ari Davidson said

Jade Lane said

James M said
I’m currently upgrading the PC tower I do my editing on and starting thinking ahead to working with Cinema DNG files. And while I’m a big fan of CalDigit and G-Technology I’m thinking I’d like to build my own RAID enclosure. 

Joe, you mentioned “the write speed is 145MB/s we estimate that at 2k 24fps the D16 will require 84MB/s.” So would a RAID 5 or RAID 6 solution with a similar sustained read/write speed do the trick? I’m even considering some USB 3.0 solutions since my laptop has the ports built in and I’m adding a USB 3.0 PCIE card to my tower.

Just curious if anyone has any previous or current DIY experiences they could share.

I’d like to pick this topic up again as its causing me some headache and confusion too.

 

I was just about to buy a G-Tech 8TB G-Raid Thunderbolt HD that in the spec states Raid 0, you mention needing a Raid 5 or 6 for the D16 CinemaDNG files and footage and also mentioned the Lacie Raid drives but their spec is not Raid 5 or 6 (unless I’m not looking properly or don’t understand).

 

Can those that actually know or understand the RAID storage situation make a list of suitable drives (USB 3 & Thunderbolt) that will be a correct solution for handling the D16 footage and at what kind of length footage over TB space storage etc.

 

I believe Raid configurations are done on the user’s end. You can use disk utility to do it (http://macperformanceguide.com…..pRAID.html)

Here’s some info on what a RAID entails: http://macperformanceguide.com…..pRAID.html

Here’s a visual representation of what your drives are doing Laugh Watercooler_63190a_1433997.jpg

 

Basically you need at least three drives for the data to be striped and allow for a failure of one drive without data loss or corruption. 

Hey Ari,

 

Thanks for the explanation, I think I understand now ? so a drive bay of some kind with three drives made up of 2TB each (6TB in total) or 4TB each (12TB in total)  partitioned correctly would work at RAID 5 or 6?

 

Glad I didn’t buy that G-Tech 8TB G-Raid Thunderbolt HD as it was two 4TB drives, which by you’re diagrams would not work, right?

March 3, 2013
2:52 pm
Ari Davidson

Camera Owner

Kickstarter Backer
Forum Posts: 379
Member Since:
March 17, 2012
Offline

Yes you can use a bay just makes sure the controller is compatible with your desired configuration. 

I’ve worked off a G-raid for a few years with a 0 configuration and it’s been just fine. I’ll probably experience a crash any day now having said that. 

 

As far as G-tech devices you should go with one of the G-Speed products, they come with a RAID 5 compatible controller. I don’t know of a product yet that interfaces via thunderbolt but sonnet makes an eSATA to Tunderbolt adapter. 

 

Happy hunting. 

March 3, 2013
10:09 pm
James M
NTX
Member
Forum Posts: 606
Member Since:
March 28, 2012
Offline
10

I’m glad people are continuing the discussion. I feel kind of silly since I didn’t update anything. What I’m doing is going with my own array. Like I said I love CalDigit products and I’ve had great luck with G-Tech but I wanted to get a bit more for my money. Though if you can afford it there are a lot of pre-configured options on the market. I’m sure some reps will even pop up here on the forums once the cameras are out in the wild. I know they did back in the early days of a certain other camera that promised 3K shooting.

I’ve found another bit of a quick rundown that explains raid to go along with the previous posts. http://www.thegeekstuff.com/20…..s-tutorial. Luckily I had some experience with older RAID systems back from a previous job and it was easy to get back into the swing of things with a little research. I decided to go with a 5 bay tower with an built in hardware controller and a separate controller card for my PC. I’ll be loading 2TB drives and setting them up in a RAID 10 array. I found this bit to explain why:

RAID 0+1 or RAID 10

RAID 0+1 or RAID 10 is a combination of RAID Levels that utilizes multiple RAID1 (mirrored) sets into a single array. Data is striped across all mirrored sets. As a comparison to RAID 5 where lower cost and fault tolerance is important, RAID 0+1 utilizes several drives to stripe data (increased performance) and then makes a copy of the striped drives to provide redundancy. Any disk can fail and no data is lost as long as the mirror of that disk is still operational. The mirrored disks eliminate the overhead and delay of parity. This level array offers high data transfer advantages of striped arrays and increased data accessibility (reads). System performance during a drive rebuild is also better than that of parity based arrays, since data does not need to be regenerated from parity information, but is copied from the other mirrored drive.

It’s a little more money but I’ve ordered drives that have longer warranties (five years) because at some point a disk will fail. Though I just couldn’t stomach the cost of full enterprise class drives. I also went with a USB 3.0/eSATA controller because eSATA is used everywhere I need to access footage and so far my external USB 3.0 drive has handled everything I can throw at it. And the 5Gbs of USB 3.0 should, by Joe’s estimates, be enough to handle CinemaDNG files.

Though if you have the money and the space inside your computer you could try just installing more drives and a beefy (i.e. expensive with battery backups) internal raid controller card. I have a couple of friends who are editors in corporate settings (lots of smaller projects but that use lots of media) and they seem to really like it. They use the onboard raid for current projects and the outboard (offboard?) raid for frequently used elements (opens, animations, music) and non-archival storage. Seems to work for them.

What options are other people using?

March 3, 2013
10:31 pm
joerubinstein

Camera Owner
Forum Posts: 1789
Member Since:
February 28, 2012
Offline

If budget permits I really like the Pegasus Promise drives:  

http://www.amazon.com/Promise-…..mise+drive

You buy one fixture and swap out 2, 3, or 4 drives for every project, raid them how ever you like.

 

Keep in mind you still need another storage solution for long term storage. Something like this:

http://www.amazon.com/HP-Ultri…..lto5+drive

 

The drive is expensive, but the media after that is dirt cheap and has some of the longest shelf life of any digital media.

March 3, 2013
10:44 pm
Thyl
Member
Forum Posts: 800
Member Since:
June 22, 2012
Offline
12

If you want to use the RAID only for backup/archiving purposes, i.e. if you will edit your footage from a different drive, and hence, only have occasional access to the RAID, you could also consider setting up a “soft RAID”, which is to say that you simply plug in several harddisks to your computer, and let the computer to the raiding. Working load on the CPU increases, but no specific device or enclosure is required.

 

With RAID 5, the total capacity of the RAID is [hard disk size of smallest HD] x ([number of hard disks] – 1). So in your given example of three hard disk of 2 TB each, you total storage is 2 TB x (3 – 1) = 4 TB.

March 4, 2013
7:49 am
James M
NTX
Member
Forum Posts: 606
Member Since:
March 28, 2012
Offline
13

I can’t afford an LTO Tape drive but I’ve worked with them in the past. And yes, if you can afford one it’s a great solution. It can read and write in the native format of many editing systems. And depending on who you believe the tapes are rated to last anywhere from 10 to 20 years. The oldest I’ve ever personally witnessed was 5 years but who knows what we’ll be saving our data to 20 years from now? Holograms maybe?

March 4, 2013
11:09 pm
Thyl
Member
Forum Posts: 800
Member Since:
June 22, 2012
Offline
14

James M said
I can’t afford an LTO Tape drive but I’ve worked with them in the past. And yes, if you can afford one it’s a great solution. It can read and write in the native format of many editing systems. And depending on who you believe the tapes are rated to last anywhere from 10 to 20 years. The oldest I’ve ever personally witnessed was 5 years but who knows what we’ll be saving our data to 20 years from now? Holograms maybe?

 

Doesn’t look to well for backup stuff. It seems that the optical media ran into a crisis. Look at the jumps. CD 650 MB -> DVD 8500 MB (factor 13) -> BD 50,000 MB (factor 6) -> BD-XL 128,000 MB (factor 2.6). And now, the focus is shifting away from media distribution to internet download. The BD is enough for 1080p at 8 or even 10 Bit per colour channel, and I strongly doubt that 4K will have a big success in the home. There had been quite some hype with new technologies in this regard a couple of years ago, including holographic storage, but no products ever came out of the labs.

 

Then, there is the cloud, and hard disks that people use for backing up their data. Archiving big date volumes is becoming a rather specialised affair, which typically means expensive. There is presently only one technology that can archive huge data for more than 10 years, LTO.

March 5, 2013
10:51 am
James M
NTX
Member
Forum Posts: 606
Member Since:
March 28, 2012
Offline

Yep. It’s a little disheartening. I was joking about the holograms. But I never thought about the smaller increases in storage. Though I suppose that’s because so much computing has been focused on average consumer users. Not necessarily media professionals. Files for most people get smaller and smaller. While shooters, editors, graphic artists and the like need more and more.

And the cloud simply unnerves me. It seems like it’s one step closer to SkyNet.

March 5, 2013
11:27 am
Thyl
Member
Forum Posts: 800
Member Since:
June 22, 2012
Offline

James M said
Yep. It’s a little disheartening. I was joking about the holograms. But I never thought about the smaller increases in storage. Though I suppose that’s because so much computing has been focused on average consumer users. Not necessarily media professionals. Files for most people get smaller and smaller. While shooters, editors, graphic artists and the like need more and more.

And the cloud simply unnerves me. It seems like it’s one step closer to SkyNet.

 

Exactly. Or at least, the data sizes with AVCHD do not increase as much as they do with 4K and raw and 12, 14, 16 Bit. Why have you been joking about holograms? I thought you were referring to this:

 

http://en.wikipedia.org/wiki/H…..atile_Disc

Evidently, yet another promising future of the past.

March 5, 2013
12:14 pm
Thyl
Member
Forum Posts: 800
Member Since:
June 22, 2012
Offline
17

You may find this dreamt-up roadmap entertaining:

 

http://www.maxellcanada.com/pd…..r_tech.pdf

 

300 GB in 2007.

March 5, 2013
2:07 pm
James M
NTX
Member
Forum Posts: 606
Member Since:
March 28, 2012
Offline
18

Okay I did find Maxell’s goals pretty funny. And I simply hadn’t kept up on the progress of holographic discs. I wish someone had cracked that one. Maybe they will eventually. And as much as I complain about hardware costs for something like an LTO drive or even a $15K holographic disc player it’s all a matter of perspective. When DVCPro HD was the hot new format a full DVCPro deck was $25,000.

Don’t get me wrong. I want the freedom of RAW shooting. I’m just building myself up for the storage and archiving investment. Though I only tend to keep the footage that is absolutely necessary. I imagine the sheer size of raw files will start to change a few habits.

March 5, 2013
11:50 pm
Thyl
Member
Forum Posts: 800
Member Since:
June 22, 2012
Offline
19

Some assumptions on optical discs:

-No holographic discs in the foreseeable future.

-No discs with more than four layers will leave the labs.

So, what can we expect? Sony and a Japanese university have developped a new 405 nm wavelength high output laser they presented mid 2010. According to Wikipedia, two layer discs could reach one terabyte. (release 2014?)

-> Four layer discs could reach two terabytes (2016?)

-> Double sided four layer discs could reach four terabytes. Makes only sense if the disc does not have to be flipped by hand, i.e. using two laser pick ups.

 

LTO will hence presumably stay at an advantage.

 

Forum Timezone: America/Los_Angeles

Most Users Ever Online: 122

Currently Online: Rabguy, jedshepherd
12 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

Razz16mm: 1624

wado1942: 1109

pask74: 863

Thyl: 800

James M: 606

iaremrsir: 498

DVCinLV: 456

Harry Lime: 448

Ari Davidson: 379

analoggab: 376

Member Stats:

Guest Posters: 1

Members: 2465

Moderators: 0

Admins: 3

Forum Stats:

Groups: 1

Forums: 8

Topics: 983

Posts: 13365

Newest Members: timedmond, prachman, ScorpioFilms, jedshepherd, theryaner, yabiku.leo, filmwomyn, joto, ekathleenc1989, NathStarr

Administrators: StenRidley: 6, elleschneider: 72, joerubinstein: 1789