WorldLink

Commoditizing Hadoop in the Cloud – It’s a Good Thing

DALLAS, TX; December 6th, 2018

This blog post will focus on Microsoft Azure technologies and discuss why most clients shouldn’t worry about underlying HDFS and Hadoop technology.

The cloud has enabled an unprecedented race towards commoditization. Data-enabling technologies are being enhanced, simplified and abstracted so that technology is easier to implement and support. One of the best examples of this process can be seen with Hadoop and the HDFS technology. In Azure today, most, if not all, of the technological capabilities HDFS and Hadoop offer are now integrated as a basic function of storage technologies. In the recent past, we would have recommended and leveraged Hadoop when we needed the following:

 

1.    A method of storing structured, semi-structured and unstructured data that was highly redundant

2.    The ability to perform batch functions on large amounts of data

Today, both of these criteria are met with capabilities that are now standard within Azure. Microsoft has made large investments into storage technology in Azure and their latest solution is Azure Data Lake Storage (ADLS) Gen2. ADLS Gen2 has become a commercially viable option for hosting a data lake, due to the following features:

  • ADLS Gen2 can be set up, deployed and configured with just a few clicks of a mouse (or as part of a DevOps activity). 
  • The data stored in ADLS Gen2 enjoys the security offered by encryption at rest, firewalls, Active Directory integration and POSIX style controls, all with unlimited scalability, hierarchical name spaces, five 9s of availability and options for seamless disaster recovery. 
  • All of these options are “built in” and do not require a significant investment of time and resources to implement and support. 

ADLS Gen2 provides deep integration with other Azure technologies, which increases the functionality that a data professional can apply to a large data set. We see many customers that use the features of ADLS Gen2 eliminate data hops and easily integrate structured, semi-structured and unstructured data into Azure SQL and Azure SQLDW as part of their larger data ecosystem. It also integrates easily with CosmosDB, PowerBI, the Azure Advanced Analytics suite and is a default destination for Azure Data Factory and Azure Databricks. 

Cost pressures are another significant area of technology commoditization. For our clients, we see that not only is the technology more powerful and easy to use, but it is also much less expensive than trying to use purpose-built solutions on premise. Hosting large data sets in ADLS Gen2 is orders of magnitude more cost effective than hosting it in dedicated Hadoop clusters, because it is optimized for cloud scale, data workloads. ADLS Gen2 also manages and tiers workload data, to help data users minimize the total cost of ownership of their data. By managing the hosting of the data and integration into other tools, ADLS Gen2 allows IT departments to deliver projects with fewer specialized technical skills, allowing them to focus on higher value business projects rather than focusing on keeping expensive technology running.  We have seen a dramatic increase in speed-to-value for clients who have been able to devote more time to value-added activities after adopting this technology approach, while managing costs in a flexible pay-as-you-go model.

 

ABOUT WORLDLINK US — WorldLink is a global technology firm that delivers three distinct areas of Solutions – Talent, Data & Analytics Consulting, and Managed Services. We provide unmatched subject matter expertise in our Talent, Cloud, and Data/Analytics practices. In addition, our focus on strategy and vision leads to strong relationships with our diverse client base. WorldLink fulfills the unique client vision instead of offering standard “out-of-the-box” solutions. We’re building what’s next. We want you to be a part of it.

by Mark Novak, Chief Architect at WorldLink

Enjoy this story? Share it to your network.

View other articles

Implementing AI Solutions: Secure, Trusted, and Customized for Your Business

Harnessing GenAI for Competitive Advantage: A Strategic Guide for Enterprises

Role of MDM in Today’s World of Big Data

Gear up for greater success with WorldLink.

Gear up for greater success with WorldLink.

Let's work together!

3880 PARKWOOD BLVD
BUILDING 2
FRISCO, TX 75034

800.673.6155

info@worldlink-us.com