Be Careful What You Automate

Part 7 in a series of posts about Data Protection as a Service. This is the first of two posts on automation… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

From breathing to paying bills, from good ideas to great habits — what’s better than automatic? Intelligent automation means never having to miss an opportunity, obligation, or requirement. When thinking about the best way to approach enterprise data backup, smart automation ranks high on our list of goals.

Automation is especially critical when dealing with the challenges of deploying limited expertise over a widening scale of mission-impacting data and the growing complexity of hybrid infrastructure. It really is a case today of sink or swim!

Inside any IT operation — but especially those concerned with availability, performance, or security — an ultimate goal really should be 100% automation. Some folks call this “autonomous operations.” Fully autonomous data protection always creeps just out of practical reach given the increasing volume of important data and emergence of new architectures and applications. Even if just to keep pace, the goal should always be to increase the level of operational automation.

Where to Start?

It is impossible to automate everything at once, so which parts should be addressed first? Automate any manual and continually repeated data protection task or responsibility, especially where human consistency (or lack thereof) affects reliability.

If someone has to stop and think about what they are doing each time they approach a repetitive task — remember key command details and gotchas, recall small steps and perform them in order, and never make a typo — there will be mistakes. When these kinds of tasks concern data protection, one or more mistakes eventually will prove quite costly to the business.

Automated Expertise

In cases in which full automation isn’t yet feasible, there is usually a smart way to automate away risk and provide intelligent, accelerating assistance. As in many IT disciplines, one can capture and encapsulate best practices, aggregate big data sets if need be, then leverage intelligent analytics to apply both ongoing learning and deep expert knowledge consistently.

Of course, it doesn’t make sense to spend a lot of effort automating mediocre (or outdated) processes; that will just deliver worse results faster. Implementation of automation best practices requires experience and expertise. One must automate not only the right things but also maintain and evolve all automation over time. It’s not a static world, and automation that is hard-coded, embedded, and forgotten can become a thorny legacy problem later on when environments change and key assumptions no longer apply.

Read about Automation and Optimization With ADP

Existing practices might very well be those relied on to just get by, and they are not necessarily considered the best. There are probably big gaps and more exceptions than we want to admit. Again, keep in mind there is little point in automating poor practices. It is important to consider that the very best source of expert automation and ongoing maintenance may not be found in-house, particularly in smaller IT shops.

…(Continued in a second post that explores automation oversight.)

It’s Not Paranoia If They Really Are Out to Get You

Part 6 in a series of posts about Data Protection as a Service… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

Ask any IT professional about enterprise data security and you can feel the tension in the room rise even before anyone starts speaking. Security is a tense topic, and for good reason. Good data security is hard. Total security today is nigh impossible.

According to the Online Trust Alliance (OTA), cybersecurity incidents targeting businesses nearly doubled from 82,000 in 2016 to 159,700 in 2017. Since the majority of cyber incidents are never reported, OTA believes the actual number in 2017 could easily have exceeded 350,000.1 Some attacks, such as ransomware, have increased by 2000% since 2015.2

In any large enterprise we can never be 100% certain that some portion of our precious corporate data can’t be hacked, corrupted or lost in some nefarious way. I’m not saying anyone is lax, lazy, or incompetent. I am saying that every day there are new emerging threats, the corporate attack surface is multiplying, and our “sensitive” data footprint is both growing and spreading.

It’s Not Paranoia If They Really Are Out to Get You

Yes Virginia, there really are evil no-good hackers! If you are any kind of company, online or not, you and your data are a big fat target!

We now have viruses that once they worm into your organization might not trip any alarms until it’s too late to prevent damage. Ransomware might slowly encrypt (or just delete/corrupt) your primary data stores. I could go on at some length about increasingly distributed attack surfaces and multiplying online touch points (e.g. kiosks, end user apps, employee mobility, IoT devices, etc.). I’ll just say that we are well beyond the time that a solid firewall was the only defense necessary. We absolutely need a more intensive “defense in depth” approach by implementing security at all levels today.

If this isn’t sounding like a fear-based approach to motivating large security improvements yet, let me pile on a bit more! Consider the modern consequences of a major data breach – your reputation will just be the first casualty. You might lose major (many or most?) customers and clients. If you fall seriously afoul of industry compliance regulations, you can be directly penalized (your fault for being hacked) up to and including losing your whole business.

Even if you don’t lose any customer data or violate privacy regulations, once your systems have been penetrated you will still need to recover to a known good state. As long as we have a trusted, protected copy of our key data kept safe, then when things do go wrong (and they will) we can quickly repair and recover.

Keeping Up is a Company Full of Jobs

So how many security experts does it take to plug all the gaps, patch all the holes, implement all the new security paradigms, and keep up with all the new threats? Even if we just look at the core defense of our data protection scheme, which must now be as close to 100% protected as possible, we have to regularly (and often immediately) patch aging backup software, ensure complete encryption of all our backup data streams and images (and don’t forget key management), automate and audit every last touch and touchpoint so we can verify systems integrity, isolate and verify data integrity (repeatedly), and of course actually and reliably backup all of our key data to start with.

Apparently, we’d need a lot of highly trained people to do this right!

For most backup products on the market today cyber security requires additional products designed to fortify the backup and cut off access to potentially vulnerable areas. The architecture may include new air-gapped landscapes run by the security team requiring the training of a new group of IT professionals, increasing the number of people involved in the process, and of course driving up costs as well.

However, I’d propose that the biggest data security risks stem from having lots of “people“ in the middle of key data protection processes. Whether through naiveté, apathy, error, or evil intent, anytime and anywhere people are involved in data protection processes there are inherent vulnerabilities.

Can we get rid of all our people? Of course not! But we can implement data protection schemes that take people out of the critical data protection equation as much as possible.

Managed Data Protection as a Service

If it takes a company of data protection experts to provide the best and most secure data protection solution possible, then so be it. But likely your company is already doing something other than being 100% internally focused on secure data protection. Is secure data protection expertise something you can just subscribe to?

Yes. You only need to find a great data protection service provider to work with – one that has a world-class security solution and a company of experts behind it. For example, Cobalt Iron offers ADP CyberShield™ which includes security features to protect your key data architected into the core data protection product. Built-in features include fully human-less backup automation inaccessible to enterprise interference, full encryption schemes, WORM policies, air-gapped and isolated landscapes for validation and recovery, and more.
If we apply the law of parsimony, also known as Occam’s Razor, then “simpler solutions are more likely to be correct than complex ones.3” In the case of protecting your backup data, a solution with security built in by design will be stronger and less expensive than ones that require plugging the holes after the fact. The security for your backup solution should not be an add on.

1. https://otalliance.org/news-events/press-releases/online-trust-alliance-reports-doubling-cyber-incidents-2017-0

2. https://economia.icaew.com/en/news/december-2017/ransomware-attacks-increase-2000-since-2015

3. https://en.wikipedia.org/wiki/Occam%27s_razor

The Rocket Science of Data Protection

Part 5 in a series of posts about Data Protection as a Service… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

At its simplest level, data protection isn’t really a hard concept. We start with a bunch of zeros and ones in a certain order and we need to ensure that regardless of disaster, interference, failure, or incompetence that we can always restore those bits to a pristine and fully operational condition. But assuring data protection in practice can be really difficult. Cascading incremental backups, complex snapshots and replication schedules, distributed data sets, increasingly mobile and demanding users, hybrid cloud operations, and motivated hackers all make complete data protection almost impossible.

Once upon a time we could just make a simple tape copy of our “main” frame and store it offsite as a backup in case we ever needed it. Those days are long gone. Today’s production data environments are complex, heterogeneous and even hybrid architectures – much of it made up of layers of virtualized infrastructure hosting increasingly agile applications. A lot of our important data no longer lives strictly within a physically defined data “center”.

What’s to be done? I propose that the future data protection answer has to consist of a broad three-pronged approach. We absolutely need high levels of automation. We need to leverage in-depth and up-to-date – almost real-time – intelligence to identify evolving threats. Then the best insight and expertise needs to be embedded at speed and scale to move operations from reactive to proactive, even to the predictive. I believe the intelligence necessary to drive all this will be encapsulated inside modern and big data-based, intelligent analytics.

All of which means that great data protection will require the equivalent of rocket science in the form of production-grade advanced intelligent analytics.

No Data Protection From Dummies

Survival for us higher life forms depends a great deal on our prediction-capable minds. We could spend our lives just reacting to given situations, but almost all of our best courses of action in any scenario depend on accurately identifying actors (good and bad) and predicting their actions, behaviors, and outcomes. It turns out data protection is much like this grand game of evolution – threats evolve, applications and usage change dynamically, and only the strong survive.

Maybe that analogy sounds a bit over the top, but my point is that enterprise data protection operations now are needing to be rolled out on a larger scale and with greater intelligence than ever before to match our increasingly digitized landscapes. Reliably protective operations need to adapt quickly to rapidly evolving threats challenging our vastly distributed and increasingly permeable “data” attack surface.

We absolutely need smarter (and faster) analytics and applications. It’s beyond time to roll in some automated intelligence into enterprise data protection operations. Such automated intelligence may come from astute automation, encapsulation of best practices, and a well-established discipline of “pattern recognition”. This pattern recognition approach has seen a new revival in the last few years for a number of reasons – larger available data sets, scalable big data algorithms, cloud computing (elastic resources on demand), and the crushing pressure to proactively identify abnormal behaviors in real-time at large scales.

Learning and applying automated expertise and intelligence will greatly enhance traditional IT operations. To be clear, we are not talking about sentient computers taking over and running IT anytime soon, but as an industry we are building “smarter” operational applications that embed increasing amounts of intelligent analytics and automated reasoning. To be really smart, we will want to learn about new threats before experiencing them ourselves! And we will want to optimize our operations based on others’ experiences as well as our own. In this respect, Service Providers have the great benefit of being able to look across the IT operations of many organizations – sometimes thousands or more – when building and training their analytically intelligent services.

A Mind In The Machine

If you are still skeptical, consider that many valuable security and protection tasks are no longer manually feasible in our current IT world. These tasks simply have to be intelligently automated. For example, imagine we want to identify security issues with data protection operations globally across thousands of data stores and access points – perhaps recognizing the abnormal signature of some new ransomware that might be slowly encrypting some vulnerable data stores. We need trained analytics that run, learn and score all of our systems 7×24, identifying intrusions as early as possible.

You don’t want (and actually can’t afford) to staff up data protection rocket scientists – there aren’t many of those out there to hire in any case. The smarter route is to engage emerging intelligent services such as those provided by Cobalt Iron. Cobalt Iron is doing more than just talking about intelligent operations, they are delivery smart, analytics-driven data protection services. They have been working diligently to embed advanced levels of expertly trained analytics into their scalable data protection offerings. They have the benefit of scale and focus – they can look across an entire “cohort” of data protection clients to distill out best practices and get ahead of emerging threats. They are data protection rocket scientists, and their “Advanced Analytics” solutions are getting smarter every day. Take advantage!

Modern Data Protection for Work and Profit

Part 4 in a series of posts about Data Protection as a Service… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

If I had to pick the two most popular IT initiatives that I hear most about today they would be automation and modernization. I find these two initiatives holding the top of the list in almost all data-centric organizations I’ve talked with. We want to be seen as modern and current, but we always want do things better, faster, and with less effort.

Automation is a natural desire – we want new capabilities enabled by new technologies, but we can’t just “add” more work to our plates every time something shiny and new comes along. To bring in the “new”, we have to either automate older solutions or converge layers of our technology stacks, which is really a form of embedding automation. The other course of action, jettisoning older solutions, usually requires “lift and shift” work to brand new platforms at some real cost and additional risk. While that may often happen (may have to happen in cases where technology has gone obsolete), volunteering for that kind of project is not usually high on anyone’s list.

To some folks automation and modernization are seen as the same thing. As time rolls forward, it’s always possible to increase automation (although not always easily) and one could argue that any kind of modernization by nature implies adding more layers of automation. In fact it’s hard to envision any kind of modernization that doesn’t significantly embed or virtualize a previously manually managed under-layer of technology.

Today, modernizing IT shops are looking at what is going on at the bright edge of IT:

  • Cloud adoption, resource commoditization, and convergence for infrastructure
  • Vastly bigger, faster (ie.. real-time) and far more distributed volumes of data
  • Growing user expectations for lower IT costs, consumer-like usage (e.g. app stores, instant provisioning) and everything-as-a-service

Interestingly, every one of these modern trends and expectations require (and are further fueled by) better ways of processing and protecting data. In order to reach significantly new speed, scale, or agility goals, data protection has to be baked in to the point where it can equivalently perform, scale, and adapt. Therefore you might guess what I’m going to say next – any significant modernization efforts must be based on highly automated data protection.

Modern Data Protection

What are some of the hallmarks of modern data protection?

  1. Proactive Policy-based Approach – If you want to be modern, you have to be proactive. And at today’s speed and scale of operations, proactive operations require policy-based approaches to automation, data protection included.
  2. Data Protection Everywhere – Data must be protected today wherever it’s generated, stored, or accessed. This means the data center, but also out to devices, edge nodes, and especially up into clouds of all kinds. Data may have gravity, but it’s also increasingly distributed. The data “center” is no longer a physical building with a secure door and raised floor.
  3. Leveraging Cloud Services – Not only is cloud real, but if you are stretching to deliver services or perform non-differentiating operations that your peers and competitors simply get by subscription from an expert-as-a-service provider, you are definitely not Modern! Time to get with the cloud program. This includes cloud storage for backup, Cloud DRaaS, and both SaaS and Management-as-a-Service solutions. You can’t beat a good service provider’s center of excellence, concentrated experience, or economies of scale.
  4. Deliver Consumer Services – There is little difference today between what internal business users want and expect from their organizations IT and what they externally want and expect as individual consumers. You need to offer Time Machine-like services, one-click provisioning, and full insulation (i.e. cloud-like) from infrastructure concerns. And keep in mind that they want to order up IT services in friendly terms like relative availability, not in terms of number of back-end backup copies, snapshot intervals, or replicate zones.
  5. Actually reduce RTO/RPO – Should go without saying, but if you can’t beat your current legacy based RTO/RPO, you probably aren’t really trying. But beyond that, you need to “show your work” with clear analytics and reports about the data being protected and the service levels actually delivered, what data is at risk, and costs in terms of showback or chargeback.

In the sense we’ve talked about in this post, IT modernization and automation may be never-ending initiatives – always part and parcel of staying abreast of new technologies that keep layering up increasing value. But there is one last angle here today that I want to point out, and that’s about finally being able to turn the corner from reactive to proactive.

If your next modernization/automation initiative can actually help you finally become proactive, you should have no qualms about getting on with it as soon as possible. And I believe that the opportunities outlined above, implemented through a data protection modernization program available by way of solutions like Cobalt Iron’s Adaptive Data Protection, can get you quickly off and running into future.

The Winner’s Problem – Technical Legacies and Legacy Technology

Part 3 in a series of posts about Data Protection as a Service… (Also posted on Cobalt Iron’s blog)

Mike Matchett, Small World Big Data

When a business is deemed successful it’s often because of great people doing awesome things with the latest technologies. Yet there is always a built-in problem for long-time winners that follows their success over time – inevitable age that causes obsolescence of both people and machinery alike.

The aging process leads to a need for ongoing retirement and refresh, even in companies that have in the past deployed prize-winning formulas. Unfortunately, winning arrangements by definition have survived, and through momentum often live well past their prime.

When it comes to data protection, what worked very well in the past to help the long surviving IT and the well-established business protect their mission-critical data most likely no longer works quite as well as it did in its prime. In fact, given the pace of change in data, applications, architectures, and even the skills and expectations in the available workforce, most organizations are working feverishly just to keep their business applications competitive. Quite often long-running back-office disciplines like data protection have to limp along as best they can, maybe adding band-aids and patches where it visibly hurts but not actually refreshing the whole solution to keep it truly current.

At least until something finally breaks in a big way – perhaps a ransom-ware attack, a rogue ex-employee, a 1000-year flood, or even full-blown compliance audit. By then of course it’s too late to be protected and prepared. The consequences can be fatal – winners can become losers overnight.

Good Legacies Can Beget Bad Ones

I see the legacy data protection challenge arising in three primary areas:

  1. Protecting legacy technology – Nothing that works well goes away fast. (Long live mainframes!) Even if users, usage, requirements, and expectations have grown and changed significantly over the years, the underlying IT methods, interface protocols, and performance capabilities of many long-successful applications and infrastructure may still be the same as the day they were first deployed – and today in 2018 that could be multiple decades past.Newer data protection architectures might require quite significant backwards integration to protect legacy technologies appropriately. And sometimes protecting both hardware and software built and deployed generations ago can still require legacy data protection technologies, doubling down on the legacy challenge.
  2. Technical legacies aging out – People grow old, especially experts it seems! Sometimes they leave even before they retire. Regardless, people inevitably grow old and age out of the workforce. And when they leave, there often aren’t equivalent knowledge replacements readily available. Old-timers just know things, particularly about legacy technologies, that no one newly available to the market will have any exposure or experience with.The learning curve for someone new to pick up legacy technology expertise may not only be steep, it may be too slippery to climb at all depending on just how legacy the technology really is. Lack of current documentation, relevant training classes, original equipment vendors, and of course senior staff mentors can all hinder effective knowledge replacement.
  3. Backup product stagnation – Many backup products have failed to evolve and keep pace with the current state of IT solutions. A partial laundry list would include virtualized servers, hyperconverged infrastructure, hybrid cloud applications, public cloud data pipelines, web applications, multi-cloud deployments, n-way replications, and globalized 7×24 operations. Let’s not even talk yet about protecting big data clusters, distributed containerized applications, temporal software defined storage, or IoT edge devices.In addition, expectations for data availability have changed significantly too – with end users more and more expecting “Apple Time Machine” like functionality in every IT application, instant RTO and seconds-level RPO from any mobile device anywhere in the world.

Even if implemented backup solutions have somewhat evolved, the necessary patches, upgrades, and migrations are likely well outside the ability of many organizations to even consider rolling out. I’m sure top of mind for many is that if a complex, legacy solution is even partly working, it’s probably best not to mess with it at all or risk blowing it up completely.

Not Just Surviving, But Thriving

So what’s the best approach to dealing with age and obsolescence? Fundamentally it’s not fighting to retain aging staff on contract into their geriatric years, or ignore the increasing wrinkles and weakening bones of your data protection program.

Rather it’s looking for a trusted service provider that specializes in data protection for enterprises like yours (like most, really). One that can afford to develop and maintain legacy technology expertise because they leverage it across multiple clients, that has current experience with most legacy hardware and software IT solutions, and that can not only maintain, but integrate, optimize and proactively operate modern data protection solutions for you on your behalf.

If you have age-related issues with your data protection, and want to keep on as a winning corporation, you might want to ask an expert data protection company like Cobalt Iron to come in and show what they can do to help keep you eternally young.