The Best Way to Manage Future IT Data Growth is to Follow These Three Steps – (1/3)

This series of posts was originally going to be about the easiest data architecture and management steps to take, but no, none of these are easy. They are hard. Very hard in fact. But quite possible necessary. Let me know what you think!

I wanted to start 2023 with some smart sounding analyst predictions, but I know by this point in my analyst career that market predictions are as about as useful as most (looking at you Gartner and IDC!), which is to say “not really.” I am a veteran military intelligence officer, an expert large datacenter performance and capacity planning consultant, and an all-around cynic (sometimes) and have learned to appreciate the impossibility of human forecasting over non-linear curves, black swans, irrational market behavior and hidden (even hostile) motivations. So rather than just predict the obvious (e.g. data is going to grow by leaps and zonga-bytes by 2025) or that people are going to spend more or less on what they need but probably more on what they want, I’m going to lay out a three step roadmap to having a chance to actually manage all those future zonga-bytes of data…

Step 1 – Future Proofing with a Storage API

The first thing to really sit down and consider is how storage is served to your storage consumers – applications and end-users. Our step one challenge is to build out an abstract data storage layer so that – at least – all of your precious underlying data can be practically managed/protected while any necessary large data migration becomes fluid and reliable without bothering end users. So, what form does abstracted storage take?

We might say you should “unify” your storage, but the concept of unification becomes very confused over all the competing vendor “unified storage” messaging. Do we want to unify on one vendor’s storage, one proprietary protocol, one type of storage or one cloud service provider? No. Not at all. Building on single foundation approaches lead to what Nassim Nicholas Taleb ¹ refers to as “fragility” (which is not a good thing!). You know not to ever put all your eggs in one damn basket!  We need to preserve and even build in the ability to more rapidly make future storage choices.

What I really mean by abstracting storage is for you to build and provide your users a logical storage consumption service, or in plain terms, serve your organization a consistent storage API. We want to be able to flex and change, mutate and evolve the underlying actual storage back-end(s) as new cloud service provider offerings, hybrid-leveraging strategies, and actually intelligent storage lifecycle solutions emerge. Even if you are a fully cloud-bound organization (although in reality most of us will be hybrid forever), you don’t want to be stuck on any one 3rd party service provider (SP) foundation stone. And watch out at every layer of your architecture – single solution “spans” can be seductive, creeping evil demons! The only single solution “span” you want to end up with is your abstract storage service.

If the idea of owning your own storage API definition sounds aggressive, it is. Do it. You need to be able to make storage cost tradeoffs and decisions. Impose the necessary cybersecurity measures. Enforce compliance and regulation. Balance ESG initiatives with performance demands. Drive storage to the edges. Squeeze clouds to get every last drop of business goodness out of them.

IT is the SP for their organizations. You are your organization’s very own internal SP so step and define what service/API you are offering and what cost/SLA tradeoffs the end user can make.

(post featured image – 2023 AI generated image from Craiyon.com)

¹ Yes, this is an affiliate link. “As an Amazon Associate I earn from qualifying purchases.” If you are interested in the books but don’t want to encourage the corruption of my independent analyst neutrality, you can follow the link but then edit the end of the URL, remove the affiliate reference and refresh the page without it.