Big data processing could be the way all data is processed

An IT industry analyst article published by SearchITOperations.


article_Big-data-processing-could-be-the-way-all-data-is-processed
Some organizations take their time with new technologies to let first adopters suffer the growing pains. But there’s no treading water in the big data stream; the current won’t wait.

Mike Matchett
Small World Big Data

Have you noticed yet? Those geeky big data platforms based on clusters of commodity nodes running open source parallel processing algorithms are evolving into some seriously advanced IT functionality.

The popular branded distributions of the Apache projects, including Hortonworks, Cloudera and MapR, are no longer simply made up of relatively basic big data batch query tools, such as Hadoop MapReduce, the way they were 10 years ago. We’ve seen advances in machine learning, SQL-based transaction support, in-memory acceleration, interactive query performance, streaming data handling, enterprise IT data governance, protection and security. And even container services, scheduling and management are on a new level. Big data platforms now present a compelling vision for the future of perhaps all IT data processing.

Wait — do I really mean all IT data center processing will be big data processing? Most of us are just getting used to the idea of investing in and building out functional data lakes to capture and collect tons of unstructured data for business intelligence tasks, offline machine learning, active archive and other secondary data applications. And many are having a hard time making those data lake initiatives successful. It’s a challenge to develop staff expertise, assure data provenance, manage metadata and master implied schemas, i.e., creating a single version of truth.

…big data isn’t just for backroom data science geeks. The technologies involved are going to define the next-generation IT data center platform…

Many organizations may be waiting for things in the big data market to settle out. Unfortunately, especially for those more comfortable being late adopters, big data processing technology development is accelerating. We see use cases rapidly proliferate, and general IT manageability of big data streams (easing adoption and integration) greatly increase.

The universal big data onslaught is not going to slow down, nor will it wait for slackers to catch up. And those able to harness their big data streams today aren’t just using them to look up old baseball stats. They are able to use data to improve and accelerate operations, gain greater competitiveness and achieve actual ROI. I’m not even going to point out the possibility that savvy big data processing will uncover new revenue opportunities and business models. Oops, just did!

If you think you are falling behind today on big data initiatives, I’d recommend you consider doubling down now. This area is moving way too fast to jump on board later and still expect to catch competitors. Big data is proving to be a huge game changer. There simply won’t be a later with big data.

I’ve written before that all data is eventually going to be big data. I’ll now add that all processing is eventually going to be big data processing. In my view, the focus of big data technology has moved from building out systems of insight over trailing big data sets to now offering ways to build convergent systems of action over all data.

In other words, big data isn’t just for backroom data science geeks. The technologies involved are going to define the next-generation IT data center platform…(read the complete as-published article there)

Serverless technology obfuscates workflows, performance data

An IT industry analyst article published by SearchITOperations.


article_Serverless-technology-obfuscates-workflows-performance-data
Serverless and microservices reshape the application stack into something that looks like a swath of stars in the sky. How do you find a slow, misconfigured component in this interconnected galaxy?

Mike Matchett
Small World Big Data

I’m hearing that IT infrastructure is dead. And who needs it anymore, really? The future is about moving up the stack to microservices and serverless technology, as we continue to abstract, embed and automate away all the complexities of explicit infrastructure layers, such as storage arrays and physical servers.

On-premises, Capex-style IT is shrinking, while rented and remotely managed hardware and cloud transformation set new standards for modern IT. All the cool kids use end-to-end orchestration, advanced machine learning, real-time management data streams, microservices architecture and insanely scalable container environments. And now we even have serverless computing, sometimes called function as a service (FaaS).

But can we have computing without the server? And where did the server go?

Serving more with serverless technology
There is a certain hazard in my life that comes from telling non-IT people that, as an IT industry analyst, I explore and explain technology. I’m asked all the time, even by my mom, questions like, “I suppose you can explain what the cloud is?

I tend to bravely charge in, and, after a lot of at-bats with this question, I’ve got the first 25 seconds down: “It’s like running all your favorite applications and storing all your data on somebody else’s servers that run somewhere else — you just rent it while you use it.” Then I lose them with whatever I say next, usually something about the internet and virtualization.

The same is mostly true with serverless computing. We are just moving one more level up the IT stack. Of course, there is always a server down in the stack somewhere, but you don’t need to care about it anymore. With serverless technology in the stack, you pay for someone else to provide and operate the servers for you.

We submit our code (functions) to the service, which executes it for us according to whatever event triggers we set. As clients, we don’t have to deal with machine instances, storage, execution management, scalability or any other lower-level infrastructure concerns.

The event-driven part is a bit like how stored procedures acted in old databases, or the way modern webpages call in JavaScript functions, hooked to and fired off in response to various clicks and other web events. In fact, AWS Lambda, a popular serverless computing service, executes client JavaScript functions, likely running Node.js in the background in some vastly scalable way.

Look ma, no server!
We need to tackle several issues to ready serverless technology for primetime enterprise use. The first is controlling complexity…(read the complete as-published article there)

What’s our future if we don’t secure IoT devices?

An IT industry analyst article published by SearchITOperations.


article_Whats-our-future-if-we-dont-secure-IoT-devices
When everything from the coffee maker to the manufacturing plant’s robots to the electric grid is connected, shouldn’t security be IT’s primary concern?

Mike Matchett
Small World Big Data

I was recently asked about the most pressing IT challenge in 2018. At first, I was going to throw out a pat answer, something like dealing with big data or finally deploying hybrid cloud architecture. But those aren’t actually all that difficult to pull off anymore.

We should be much more afraid of today’s human ignorance than tomorrow’s AI.

Then I thought about how some people like to be irrationally scared about the future, and bogeyman like artificial intelligence in particular. But AI really isn’t the scary part. It’s the blind trust we already tend to put into black-box algorithms and short-sighted local optimizations that inevitably bring about unintended consequences. We should be much more afraid of today’s human ignorance than tomorrow’s AI.

Instead, what I came up with as the hard, impending problem for IT is how to adequately secure the fast-expanding internet of things. To be clear, I interpret IoT rather broadly to include existing mobile devices — e.g., smartphones that can measure us constantly with multiple sensors and GPS — connected consumer gadgets and household items, and the burgeoning realm of industrial IoT.

The rush to secure IoT devices isn’t just about your personal things, as in the risk of someone hacking your future driverless car. The potential scope of an IoT security compromise is, by definition, huge. Imagine every car on the road hacked — at the same time.

IoT exploits could also go wide and deep. Sophisticated compromises could attack your car, your phone, your home security system, your pacemaker and your coffeepot simultaneously. Imagine every coffee machine out of service on the same morning. We haven’t even begun to outline the potential nightmare scenarios caused by insecure IoT devices. And I sure hope Starbucks is keeping some analog percolators on standby.

If personal physical danger isn’t scary enough, think about the ease with which a single penetration of a key connected system could cause a nationwide or even global disaster. For example, a 2003 cascading power outage that affected over 50 million people in New England was triggered by a single alarm system misconfiguration. An inability to recover or reset something easily at that scale could push one into imagining a truly dystopian future.

Vulnerable with a capital V
What worries me more than the possibility of a large, direct attack is the very real likelihood of slow, insidious, creeping subversion, achieved through IoT device security breaches. And not just by one party or a single bad actor, but by many competing interests and organizations over time — some with supposedly good intentions.

We will make mistakes, take shortcuts and ignore vulnerabilities until it’s too late.

The total IoT attack surface will be too large to keep everything fully secured…(read the complete as-published article there)

Reap IT automation benefits in every layer of the stack

An IT industry analyst article published by SearchITOperations.


article_Reap-IT-automation-benefits-in-every-layer-of-the-stack
Automation technologies create an artificial brain for IT operations, but that won’t turn skilled admins and engineers into zombies — far from it.

Mike Matchett
Small World Big Data

As a technology evangelist and professional IT systems optimizer, I see the benefits of IT automation and can only champion trends that increase it. When we automate onerous tasks and complex manual procedures, we naturally free up time to focus our energies higher in the stack. Better and more prevalent automation increases the relative return on our total effort so that we each become more productive and valuable. Simply put, IT automation provides leverage. So it’s all good, right?

Another IT automation benefit is that it captures, encapsulates and applies valuable knowledge to real-world problems. And actually, it’s increasingly hard to find IT automation platforms that don’t promote embedded machine learning and artificially intelligent algorithms. There is a fear that once our hard-earned knowledge is automated, we’ll no longer be necessary.

So, of course, I need to temper my automation enthusiasm. Automation can eliminate low-level jobs, and not everyone can instantly adjust or immediately convert to higher-value work. For example, industrial robots, self-driving cars or a plethora of internet of things (IoT)-enabled devices that cut out interactions with local retailers all tend to remove the bottom layer of the related pyramid of available jobs. In those situations, there will be fewer, more-utilized positions left as one climbs upward in skill sets.

Still, I believe automation, in the long run, can’t help but create even more pyramids to climb. We are a creative species after all. Today, we see niches emerging for skilled folks with a combination of internal IT and, for example, service provider, high-performance computing, data science, IoT and DevOps capabilities.

Automation initiatives aren’t automatic

If one squints a bit, almost every IT initiative aims to increase automation.

A service provider has a profit motive, so the benefit of IT automation is creating economies of scale. Those, in turn, drive competitive margins. But even within enterprise IT, where IT is still booked as a cost center, the drive toward intelligent automation is inevitable. Today, enterprise IT shops, following in the footsteps of the big service providers, are edging toward hybrid cloud-scale operations internally and finding that serious automation isn’t a nice-to-have, but a must-have.If one squints a bit, almost every IT initiative aims to increase automation. Most projects can be sorted roughly into these three areas with different IT automation benefits, from cost savings to higher uptime:

  • Assurance. Efforts to automate support and help desk tasks, shorten troubleshooting cycles, shore up security, protect data, reduce outages and recover operations quickly.
  • Operations. Necessary automation to stand up self-service catalogs, provision apps and infrastructure across hybrid and multi-cloud architectures to enable large-scale operations, and orchestrate complex system management tasks.
  • Optimization. Automation that improves or optimizes performance in complex, distributed environments, and minimizes costs through intelligent brokering, resource recovery and dynamic usage balancing.

Automation enablers at large
Successful automation initiatives don’t necessarily start by implementing new technologies like machine learning or big data. Organizational commitment to automation can drive a whole business toward a new, higher level of operational excellence…(read the complete as-published article there)

A serverless architecture could live in your data center

An IT industry analyst article published by SearchITOperations.


article_A-serverless-architecture-could-live-in-your-data-center
Just because you don’t see the server doesn’t mean it’s not there. Serverless frameworks are superseding containers, but is the extra abstraction worth it?

Mike Matchett

Have you figured out everything you need to know about managing and operating container environments already? How to host them in your production data centers at scale? Transform all your legacy apps into containerized versions? Train your developers to do agile DevOps, and turn your IT admins into cloud brokers? Not quite yet?

I hate to tell you, but the IT world is already moving past containers. Now you need to look at the next big thing: serverless computing.

I don’t know who thought it was a good idea to label this latest application architecture trend serverless computing. Code is useless, after all, unless it runs on a computer. There has to be a server in there somewhere. I guess the idea was to imply that when you submit application functionality for execution without caring about servers, it feels completely serverless.

In cloud infrastructure as a service, you don’t have to own or manage your own physical infrastructure. With cloud serverless architecture, you also don’t have to care about virtual machines, operating systems or even containers.

Serving more through serverless architecture?

So what is serverless computing? It’s a service in which a programmer can write relatively contained bits of code and then directly deploy them as standalone, function-sized microservices. You can easily set up these microservices to execute on a serverless computing framework, triggering or scheduling them by policy in response to supported events or API calls.

A serverless framework is designed to scale well with inherently stateless microservices — unlike today’s containers, which can host stateful computing as well as stateless code. You might use serverless functions to tackle applications that need highly elastic, event-driven execution or when you create a pipeline of arbitrary functionality to transform raw input into polished output. This event-pipeline concept meshes well with expected processing needs related to the internet of things. It could also prove useful with applications running in a real-time data stream.

A well-known public cloud example of serverless computing is Amazon Web Service’s Lambda. The Lambda name no doubt refers to anonymous lambda functions used extensively in functional programming. In languages such as JavaScript or Ruby, a function can be a first-class object defined as a closure of some code function within a prescribed variable scope. Some languages have actual lambda operators that a programmer can use to dynamically create new function objects at runtime (e.g., as other code executes).

So with a serverless framework, where does the actual infrastructure come into the picture? It’s still there, just under multiple layers of abstraction. Talk about software-defined computing. With this latest evolution into serverless computing, we now have perhaps several million lines of system- and platform-defining code between application code and hardware. It’s a good thing Moore’s Law hasn’t totally quit on us…(read the complete as-published article there)