Be Careful What You Automate – part 2

Shining a Light on Automation Oversight

Part 8 in a series of posts about Data Protection as a Service. This is the second of two posts on automation… (Also posted on Cobalt Iron’s blog)

The first part of this series explored the need for automation in modern backup solutions. Not just automating existing operations, but seeking out and demanding best practices to protect enterprise business.

Mike Matchett, Small World Big Data

When thinking about the best way to approach enterprise data backup, automation ranks high on the list of goals. Automation means consistency, reliability, and out of sight, out of mind — and this can be a dangerous position.

Automation Oversight

Automation without oversight is a recipe for failure at scale. The larger the operation that is automated, the more checks and balances in the form of monitoring and quality/integrity checks are required. Monitoring goes hand in hand with smart automation (and sometimes provides built-in optimization feedback). It must be built-in to help identify when things don’t fully compute, when scripts can’t run, and when and where protection coverage isn’t complete.

Data protection automation at scale also needs to be securely operated. Perhaps even more securely than the data it’s operating over. If automation processes can be hijacked or hoodwinked, recovery may be impossible.

The solution to achieving great data protection coverage — faster, better, and more cost-effective — through automation lies in engaging with companies that have deep technical expertise, have years of experience embedding best practices for applications at scale, and can truly offer intelligent data protection automation. More than that, there is a growing need for a trusted partner that is in business to build and evolve even better automated data protection over time.

Examples of automation with oversight might include integration of the backup solution with enterprise orchestration tools such as ServiceNow and Remedy to ensure that backup is receiving the same level of visibility as other critical business operations. A second example would be a solution that automatically detects and provisions backup services for newly created virtual machines. Another example would be a solution that proactively resolves problems by leveraging the power of analytics.


One point of view is that IT automation is really about stress reduction, even though most people talk about the beneficial impacts in terms of process efficiency, risk and cost reduction, and even service assurance. This line of thinking about automation advocates that if people are not sitting directly in the critical path, then they can’t disrupt the operation, intentionally or unintentionally.

To be clear, this is not suggesting that people should be taken out of IT. Rather, that when critical recurring jobs just get done automatically — the backups run, the data is validated, and all the systems are protected — smart people can move on to bigger and better uses of their time and creative energy.

So really, what’s better than automating backup best practices, smartly AND securely? If that can be accomplished, rather than just repetitively running through spotty procedures, IT professionals will discover time to innovate and add new value to the business.

Be Careful What You Automate

Part 7 in a series of posts about Data Protection as a Service. This is the first of two posts on automation… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

From breathing to paying bills, from good ideas to great habits — what’s better than automatic? Intelligent automation means never having to miss an opportunity, obligation, or requirement. When thinking about the best way to approach enterprise data backup, smart automation ranks high on our list of goals.

Automation is especially critical when dealing with the challenges of deploying limited expertise over a widening scale of mission-impacting data and the growing complexity of hybrid infrastructure. It really is a case today of sink or swim!

Inside any IT operation — but especially those concerned with availability, performance, or security — an ultimate goal really should be 100% automation. Some folks call this “autonomous operations.” Fully autonomous data protection always creeps just out of practical reach given the increasing volume of important data and emergence of new architectures and applications. Even if just to keep pace, the goal should always be to increase the level of operational automation.

Where to Start?

It is impossible to automate everything at once, so which parts should be addressed first? Automate any manual and continually repeated data protection task or responsibility, especially where human consistency (or lack thereof) affects reliability.

If someone has to stop and think about what they are doing each time they approach a repetitive task — remember key command details and gotchas, recall small steps and perform them in order, and never make a typo — there will be mistakes. When these kinds of tasks concern data protection, one or more mistakes eventually will prove quite costly to the business.

Automated Expertise

In cases in which full automation isn’t yet feasible, there is usually a smart way to automate away risk and provide intelligent, accelerating assistance. As in many IT disciplines, one can capture and encapsulate best practices, aggregate big data sets if need be, then leverage intelligent analytics to apply both ongoing learning and deep expert knowledge consistently.

Of course, it doesn’t make sense to spend a lot of effort automating mediocre (or outdated) processes; that will just deliver worse results faster. Implementation of automation best practices requires experience and expertise. One must automate not only the right things but also maintain and evolve all automation over time. It’s not a static world, and automation that is hard-coded, embedded, and forgotten can become a thorny legacy problem later on when environments change and key assumptions no longer apply.

Read about Automation and Optimization With ADP

Existing practices might very well be those relied on to just get by, and they are not necessarily considered the best. There are probably big gaps and more exceptions than we want to admit. Again, keep in mind there is little point in automating poor practices. It is important to consider that the very best source of expert automation and ongoing maintenance may not be found in-house, particularly in smaller IT shops.

…(Continued in a second post that explores automation oversight.)

It’s Not Paranoia If They Really Are Out to Get You

Part 6 in a series of posts about Data Protection as a Service… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

Ask any IT professional about enterprise data security and you can feel the tension in the room rise even before anyone starts speaking. Security is a tense topic, and for good reason. Good data security is hard. Total security today is nigh impossible.

According to the Online Trust Alliance (OTA), cybersecurity incidents targeting businesses nearly doubled from 82,000 in 2016 to 159,700 in 2017. Since the majority of cyber incidents are never reported, OTA believes the actual number in 2017 could easily have exceeded 350,000.1 Some attacks, such as ransomware, have increased by 2000% since 2015.2

In any large enterprise we can never be 100% certain that some portion of our precious corporate data can’t be hacked, corrupted or lost in some nefarious way. I’m not saying anyone is lax, lazy, or incompetent. I am saying that every day there are new emerging threats, the corporate attack surface is multiplying, and our “sensitive” data footprint is both growing and spreading.

It’s Not Paranoia If They Really Are Out to Get You

Yes Virginia, there really are evil no-good hackers! If you are any kind of company, online or not, you and your data are a big fat target!

We now have viruses that once they worm into your organization might not trip any alarms until it’s too late to prevent damage. Ransomware might slowly encrypt (or just delete/corrupt) your primary data stores. I could go on at some length about increasingly distributed attack surfaces and multiplying online touch points (e.g. kiosks, end user apps, employee mobility, IoT devices, etc.). I’ll just say that we are well beyond the time that a solid firewall was the only defense necessary. We absolutely need a more intensive “defense in depth” approach by implementing security at all levels today.

If this isn’t sounding like a fear-based approach to motivating large security improvements yet, let me pile on a bit more! Consider the modern consequences of a major data breach – your reputation will just be the first casualty. You might lose major (many or most?) customers and clients. If you fall seriously afoul of industry compliance regulations, you can be directly penalized (your fault for being hacked) up to and including losing your whole business.

Even if you don’t lose any customer data or violate privacy regulations, once your systems have been penetrated you will still need to recover to a known good state. As long as we have a trusted, protected copy of our key data kept safe, then when things do go wrong (and they will) we can quickly repair and recover.

Keeping Up is a Company Full of Jobs

So how many security experts does it take to plug all the gaps, patch all the holes, implement all the new security paradigms, and keep up with all the new threats? Even if we just look at the core defense of our data protection scheme, which must now be as close to 100% protected as possible, we have to regularly (and often immediately) patch aging backup software, ensure complete encryption of all our backup data streams and images (and don’t forget key management), automate and audit every last touch and touchpoint so we can verify systems integrity, isolate and verify data integrity (repeatedly), and of course actually and reliably backup all of our key data to start with.

Apparently, we’d need a lot of highly trained people to do this right!

For most backup products on the market today cyber security requires additional products designed to fortify the backup and cut off access to potentially vulnerable areas. The architecture may include new air-gapped landscapes run by the security team requiring the training of a new group of IT professionals, increasing the number of people involved in the process, and of course driving up costs as well.

However, I’d propose that the biggest data security risks stem from having lots of “people“ in the middle of key data protection processes. Whether through naiveté, apathy, error, or evil intent, anytime and anywhere people are involved in data protection processes there are inherent vulnerabilities.

Can we get rid of all our people? Of course not! But we can implement data protection schemes that take people out of the critical data protection equation as much as possible.

Managed Data Protection as a Service

If it takes a company of data protection experts to provide the best and most secure data protection solution possible, then so be it. But likely your company is already doing something other than being 100% internally focused on secure data protection. Is secure data protection expertise something you can just subscribe to?

Yes. You only need to find a great data protection service provider to work with – one that has a world-class security solution and a company of experts behind it. For example, Cobalt Iron offers ADP CyberShield™ which includes security features to protect your key data architected into the core data protection product. Built-in features include fully human-less backup automation inaccessible to enterprise interference, full encryption schemes, WORM policies, air-gapped and isolated landscapes for validation and recovery, and more.
If we apply the law of parsimony, also known as Occam’s Razor, then “simpler solutions are more likely to be correct than complex ones.3” In the case of protecting your backup data, a solution with security built in by design will be stronger and less expensive than ones that require plugging the holes after the fact. The security for your backup solution should not be an add on.




The Rocket Science of Data Protection

Part 5 in a series of posts about Data Protection as a Service… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

At its simplest level, data protection isn’t really a hard concept. We start with a bunch of zeros and ones in a certain order and we need to ensure that regardless of disaster, interference, failure, or incompetence that we can always restore those bits to a pristine and fully operational condition. But assuring data protection in practice can be really difficult. Cascading incremental backups, complex snapshots and replication schedules, distributed data sets, increasingly mobile and demanding users, hybrid cloud operations, and motivated hackers all make complete data protection almost impossible.

Once upon a time we could just make a simple tape copy of our “main” frame and store it offsite as a backup in case we ever needed it. Those days are long gone. Today’s production data environments are complex, heterogeneous and even hybrid architectures – much of it made up of layers of virtualized infrastructure hosting increasingly agile applications. A lot of our important data no longer lives strictly within a physically defined data “center”.

What’s to be done? I propose that the future data protection answer has to consist of a broad three-pronged approach. We absolutely need high levels of automation. We need to leverage in-depth and up-to-date – almost real-time – intelligence to identify evolving threats. Then the best insight and expertise needs to be embedded at speed and scale to move operations from reactive to proactive, even to the predictive. I believe the intelligence necessary to drive all this will be encapsulated inside modern and big data-based, intelligent analytics.

All of which means that great data protection will require the equivalent of rocket science in the form of production-grade advanced intelligent analytics.

No Data Protection From Dummies

Survival for us higher life forms depends a great deal on our prediction-capable minds. We could spend our lives just reacting to given situations, but almost all of our best courses of action in any scenario depend on accurately identifying actors (good and bad) and predicting their actions, behaviors, and outcomes. It turns out data protection is much like this grand game of evolution – threats evolve, applications and usage change dynamically, and only the strong survive.

Maybe that analogy sounds a bit over the top, but my point is that enterprise data protection operations now are needing to be rolled out on a larger scale and with greater intelligence than ever before to match our increasingly digitized landscapes. Reliably protective operations need to adapt quickly to rapidly evolving threats challenging our vastly distributed and increasingly permeable “data” attack surface.

We absolutely need smarter (and faster) analytics and applications. It’s beyond time to roll in some automated intelligence into enterprise data protection operations. Such automated intelligence may come from astute automation, encapsulation of best practices, and a well-established discipline of “pattern recognition”. This pattern recognition approach has seen a new revival in the last few years for a number of reasons – larger available data sets, scalable big data algorithms, cloud computing (elastic resources on demand), and the crushing pressure to proactively identify abnormal behaviors in real-time at large scales.

Learning and applying automated expertise and intelligence will greatly enhance traditional IT operations. To be clear, we are not talking about sentient computers taking over and running IT anytime soon, but as an industry we are building “smarter” operational applications that embed increasing amounts of intelligent analytics and automated reasoning. To be really smart, we will want to learn about new threats before experiencing them ourselves! And we will want to optimize our operations based on others’ experiences as well as our own. In this respect, Service Providers have the great benefit of being able to look across the IT operations of many organizations – sometimes thousands or more – when building and training their analytically intelligent services.

A Mind In The Machine

If you are still skeptical, consider that many valuable security and protection tasks are no longer manually feasible in our current IT world. These tasks simply have to be intelligently automated. For example, imagine we want to identify security issues with data protection operations globally across thousands of data stores and access points – perhaps recognizing the abnormal signature of some new ransomware that might be slowly encrypting some vulnerable data stores. We need trained analytics that run, learn and score all of our systems 7×24, identifying intrusions as early as possible.

You don’t want (and actually can’t afford) to staff up data protection rocket scientists – there aren’t many of those out there to hire in any case. The smarter route is to engage emerging intelligent services such as those provided by Cobalt Iron. Cobalt Iron is doing more than just talking about intelligent operations, they are delivery smart, analytics-driven data protection services. They have been working diligently to embed advanced levels of expertly trained analytics into their scalable data protection offerings. They have the benefit of scale and focus – they can look across an entire “cohort” of data protection clients to distill out best practices and get ahead of emerging threats. They are data protection rocket scientists, and their “Advanced Analytics” solutions are getting smarter every day. Take advantage!

Modern Data Protection for Work and Profit

Part 4 in a series of posts about Data Protection as a Service… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

If I had to pick the two most popular IT initiatives that I hear most about today they would be automation and modernization. I find these two initiatives holding the top of the list in almost all data-centric organizations I’ve talked with. We want to be seen as modern and current, but we always want do things better, faster, and with less effort.

Automation is a natural desire – we want new capabilities enabled by new technologies, but we can’t just “add” more work to our plates every time something shiny and new comes along. To bring in the “new”, we have to either automate older solutions or converge layers of our technology stacks, which is really a form of embedding automation. The other course of action, jettisoning older solutions, usually requires “lift and shift” work to brand new platforms at some real cost and additional risk. While that may often happen (may have to happen in cases where technology has gone obsolete), volunteering for that kind of project is not usually high on anyone’s list.

To some folks automation and modernization are seen as the same thing. As time rolls forward, it’s always possible to increase automation (although not always easily) and one could argue that any kind of modernization by nature implies adding more layers of automation. In fact it’s hard to envision any kind of modernization that doesn’t significantly embed or virtualize a previously manually managed under-layer of technology.

Today, modernizing IT shops are looking at what is going on at the bright edge of IT:

  • Cloud adoption, resource commoditization, and convergence for infrastructure
  • Vastly bigger, faster (ie.. real-time) and far more distributed volumes of data
  • Growing user expectations for lower IT costs, consumer-like usage (e.g. app stores, instant provisioning) and everything-as-a-service

Interestingly, every one of these modern trends and expectations require (and are further fueled by) better ways of processing and protecting data. In order to reach significantly new speed, scale, or agility goals, data protection has to be baked in to the point where it can equivalently perform, scale, and adapt. Therefore you might guess what I’m going to say next – any significant modernization efforts must be based on highly automated data protection.

Modern Data Protection

What are some of the hallmarks of modern data protection?

  1. Proactive Policy-based Approach – If you want to be modern, you have to be proactive. And at today’s speed and scale of operations, proactive operations require policy-based approaches to automation, data protection included.
  2. Data Protection Everywhere – Data must be protected today wherever it’s generated, stored, or accessed. This means the data center, but also out to devices, edge nodes, and especially up into clouds of all kinds. Data may have gravity, but it’s also increasingly distributed. The data “center” is no longer a physical building with a secure door and raised floor.
  3. Leveraging Cloud Services – Not only is cloud real, but if you are stretching to deliver services or perform non-differentiating operations that your peers and competitors simply get by subscription from an expert-as-a-service provider, you are definitely not Modern! Time to get with the cloud program. This includes cloud storage for backup, Cloud DRaaS, and both SaaS and Management-as-a-Service solutions. You can’t beat a good service provider’s center of excellence, concentrated experience, or economies of scale.
  4. Deliver Consumer Services – There is little difference today between what internal business users want and expect from their organizations IT and what they externally want and expect as individual consumers. You need to offer Time Machine-like services, one-click provisioning, and full insulation (i.e. cloud-like) from infrastructure concerns. And keep in mind that they want to order up IT services in friendly terms like relative availability, not in terms of number of back-end backup copies, snapshot intervals, or replicate zones.
  5. Actually reduce RTO/RPO – Should go without saying, but if you can’t beat your current legacy based RTO/RPO, you probably aren’t really trying. But beyond that, you need to “show your work” with clear analytics and reports about the data being protected and the service levels actually delivered, what data is at risk, and costs in terms of showback or chargeback.

In the sense we’ve talked about in this post, IT modernization and automation may be never-ending initiatives – always part and parcel of staying abreast of new technologies that keep layering up increasing value. But there is one last angle here today that I want to point out, and that’s about finally being able to turn the corner from reactive to proactive.

If your next modernization/automation initiative can actually help you finally become proactive, you should have no qualms about getting on with it as soon as possible. And I believe that the opportunities outlined above, implemented through a data protection modernization program available by way of solutions like Cobalt Iron’s Adaptive Data Protection, can get you quickly off and running into future.