The Advent of Big Data

(Excerpt from original post on the Taneja Group News Blog)


The advent of the era of “Big Data” presents a real new opportunity to accelerate business, improve government, and even enrich our personal lives in ways we have just begun imagining.  With new methods and technology we can now feasibly mine valuable insights and knowledge out of huge mountains of raw data. At the same time, implementing big data projects is a challenge not just to the brave new world data scientists boldly exploring new algorithms and data sources, but perhaps even more so to the IT professionals tasked with implementing real world production-quality big data platforms and solutions.

Explosive data growth has been a perennial problem for IT well before big data requirements came along.  We’ve seen technologies like dedupe and compression become common features in enterprise storage, and yet the data deluge keeps coming.  As the world moves into the 21st century, evolving and emerging technologies increasingly instrument, measure and record finer grained details at higher volumes of ever more subjects.  In the business world this means vastly more raw data is being generated every day about customers (and their behavior both on and off-line), vendors and supply chains, and internal operations and processes.  

And IT is having to rethink data life cycle management because one of the tenets of big data analysis is that you have to retain all that big data (in its original raw form according to some best practices) in order to analyze it.  Data deletion may become a thing of the past in favor of near infinite archives – and that certainly creates tension with risk mitigation policies that limit certain kinds of data lifetimes (i.e. email). Archiving itself takes on a whole new meaning when archives also need to be actively accessible for historical analysis.

Just capturing and maintaining the increasing stream of “3-V’s” (huge Volume, wide Variety, high Velocity) big data is challenge enough, but IT also needs to support timely analysis.  This entails standing up (or subscribing to) clustered Hadoop and other advanced BI stacks that still at times seem more like university projects than commercial solutions.  Hot new web companies with VC money to burn can write their own big data solutions, but even the most agile enterprise IT shop will face support challenges implementing HPC-like solutions for production big data applications.

And let’s not forget that big data analyses may also produce some fairly big results that need to be searchable and shareable across often globally distributed organizations in order to actually recognize value from the effort.  Cloud storage is getting cheaper and more secure by the day, and WAN optimization technologies more efficient, but data transfer constraints and costs will still bottleneck ever larger big data workflows.

Big data and the potential benefits are tremendously exciting, and the question isn’t really if you should pursue big data projects, but can you?  IT organizations will need new skills, cutting edge resources, and most of all business commitment to look for new intelligence in their big data to competitively enhance existing processes and derive new business opportunities. Big data is a chance to evolve through applied IT, but only if the business organization understands and aligns to the opportunity.

…(read the full post)

Is Cloud Content Collaboration Safe? – Huddle Gets Security Nod From U.S. Gov

(Excerpt from original post on the Taneja Group News Blog)

Companies are worried about corporate data being uploaded by employees to personal public cloud storage and collaboration sites like Dropbox or Google Docs. There is no doubt that users want online content collaboration that makes file-sharing as easy as drag and drop. At the same time, IT needs to retain full data governance including controls for access, audit, and locatabilty (and often assure compliance).  And while there are a number of ways for companies to build and offer their own internal “secure” private file sharing clouds like VMware’s Project Octopus, not to mention full-blown MS Sharepoint, these put IT in the position of building and operating internal services that are challenged to match the easy usabilty, “everywhere” accessibility, and competitive cost of online SaaS solutions. The results of internally built collaboration projects are often as frustrating to manage for IT as they are for their users to use (and then more data sneaks online).

We met with fast-growing content collaboration service provider Huddle recently and came away impressed with the security hurdles they’ve managed to leap with their online collaboration platform, garnering not only validation from the U.K. government, but now also with the U.S. by way of a partnership with In-Q-Tel, the non-profit that invests in R+D on behalf of U.S. intelligence agencies. If they can help Homeland Security share content online safely, securely, and even intelligently with the kinds of collaboration features that corporate business warriors demand on all their devices these days, then enterprise IT shops are going to quickly take notice.  

Huddle claims to:

…(read the full post)