Myths surrounding big data technology

An IT industry analyst article published by SearchStorage.

Big data technology is a big deal for storage shops, and a clear understanding of what it means is required to configure storage for big data apps.

I love the idea of changing the world through big data technology. Big data promises we’ll all be IT superheroes just by storing more raw data than ever before and then using parallel processing techniques to yield great new insights that will catapult our company to the top. Good storage is costly and the rate that interesting new data is produced increases daily, but the Apache Hadoop product calls for leveraging scale-out commodity server nodes with cheap local disk.

Of course, there’s more to it. Conceptually, big data products bring new ways to store and analyze the mountains of data that we used to discard. There’s certainly information and insight to be mined, but the definitions are fuzzy, the hype is huge and the mining technologies themselves are still rapidly evolving.

Adding to the confusion, big data technology has been enthusiastically marketed by just about every storage vendor on the planet. But despite the marketing, I believe it’s just a matter of time before every competitive IT shop has a real big-data solution to implement or manage, if only because of staggering data growth. For those just setting out on a big data journey, watch out for these common myths.

Myth No. 1: Just do it

A sure way to waste a lot of money is to aggregate tons of data on endlessly scalable clusters and hope that your star data scientist will someday discover the hidden keys to eternal profit.

To succeed with any IT project, big data included, you need to have a business value proposition in mind and an achievable plan laid out. Research is good and those “aha” moments can be exciting, but by the time big data gets to IT, there needs to be a more practical goal than just a desire to “see what might be in there.”

Myth No. 2: Store everything

…(read the complete as-published article there)