The Best Way to Manage Future IT Data Growth is to Follow These Three Steps – (3/3)
This series of posts was originally going to be about the easiest data architecture and management steps to take, but no, none of these are easy. They are hard. Very hard in fact. But quite possible necessary. Let me know what you think!
The third step in our managing data growth roadmap!
(Step 1 – Future Proofing with a Storage API)
(Step 2 – Manage Data by its Value)
Step 3 – Push Compute to the Data Edge
Engage with and enable more distributed and edgier computing models. While addressing steps 1+2 (building out your storage service and automating value-based (values-based?) data management), you’ll probably find that you’ll need to push processing out of the data “center” towards the edges of your data universe. Looking around today, containerization, 5G/mobile/edge, and edge/embeddable AI/ML capabilities are ripe for deployment. We are quickly going to see compute functions pushed out towards the edges of our systems where the data is rushing in the fastest.
Take advantage of the distribution provided by those capabilities to distribute data services, data management and data security tasks. A centralized brain will never be good enough. Even a cloud-hosted “center” will become too expensive, too vaporously managed to contain it all. Use the cloud for elasticity, for bursting, for brokering the cheapest building stones, even for managing distributed edges. But the way to really get ahead of the tsunami of data coming in is to meet it out towards the edge.
Only aggregatable data should be moved centrally, and it should be aggregated at the first best point where that data comes together. We then need to push compute tasks out towards the specific data each functionality needs where it makes the most sense – where the lowest detail of detail required for that functionality is readily accessible. This is a hyperdynamic IT infrastructure that will actively minimize storage costs while ensuring both protection/compliance requirements are met and SLA’s are exceeded.
The operational complexity could be staggering for humans to manage, but I firmly believe we are already in the presence of smart-enough AI/ML solutions that could readily optimize a complex web of compute and data migration. AI/ML solutions would only get better as they learned new tricks over time. That’s future-proofing!
(post featured image – 2023 AI generated image from Craiyon.com)