How not to store data
This blog has been spending a lot of time discussing storage, and will continue to do so. This is, after all, perhaps the largest new workflow challenge with HD, 2K and 4K cameras, for those moving up from the SD world. I’ve already discussed archiving (offline storage) a bit, and I’m planning some posts on online storage. But first, I think a word on how not to store data might be in order.
First off, just to be clear, since there’s some terminology confusion… in the production world, “online” storage doesn’t mean Internet or network storage. It means specifically what we could call “active” storage, almost always hard drives. The drives you actually edit off off, as contrasted with, for instance, tape or DVD backups sitting in your closet, which are “offline” storage.
Onward.
Digital production involves working with large amounts of data. The IT industry has developed a wealth of techniques for managing data in ways that are robust, affordable, and convenient.
Unfortunately, many smaller digital production facilities don’t really have a serious IT guy on staff. Their data management schemes tend to resemble what one would expect if a technique that works well for hobbyists — sticking your footage on external Firewire drives — is scaled up for the entire business. I’ve seen facilities with a dozen or more external drives kicking around, with data haphazardly distributed around on them, and with no set of practices in place to make sure that there are always at least two copies of everything.
You really don’t want to do this. It’s expensive. It wastes a lot of time. And when you’re dealing with RED footage (estimate about one hour per 100 GB with REDCODE RAW 4K), it’s going to get really impractical, because you’ll need multiple external drives per feature.
Far worse, from the perspective of someone with IT experience, who knows how to keep data safe… it’s really just downright scary.
There are far better options. I’ll be posting about some of them this week.
Chris, I just found your blog and I’m very interested in what your solutions are to the above post.
“There are far better options. I’ll be posting about some of them this week.”
Did I miss them? I’m not see better options anywhere.
Yeah, that post got a little sidetracked. My most recent post (on ZFS) is a sort of prelude to that. The storage post should be up in a couple of days.
[...] This post is a follow-up to a post from a couple of weeks ago, “How not to store data“, in which I admonished against the all-too-common practice at small production facilities of stashing data on lots of external hard drives with no coherent management plan. [...]