Welcome!

Microsoft Cloud Authors: Nick Basinger, Kevin Benedict, Pat Romanski, Liz McMillan, Lori MacVittie

Related Topics: Containers Expo Blog, Microservices Expo

Containers Expo Blog: Article

The EDW Is Dead! Is Data Virtualization the Crown Prince?

New challenges threaten the reign of enterprise data warehousing

"The King is dead. Long live the King" is a traditional proclamation made following the accession of a new monarch.   But what happens when the original king was highly revered and his successor brings foreign ways to the throne?

A Controversial Statement by Bloor Research
Philip Howard, noted IT researcher at Bloor Research, recently published an IT-Director article entitled The EDW is dead where he declares the well regarded monarch of data integration deceased.   In this article he identifies two major flaws in the traditional enterprise data warehouse approach that support his controversial stand.

Flaw One - Multiple Data Types
The first flaw in the enterprise data warehouse concept relates to how data is stored and processed.   Not all data is structured.  Unstructured (for example text and images) and streaming (for example web clicks and sensor data) are also critical and growing elements in the enterprise data landscape.   According to Howard, "It is just about possible to conceive of a platform that supports relational, Hadoop-like and streaming data at some point in the future but it's not going to happen soon-if it ever does.  So, certainly for the time being, there is no prospect of an EDW supporting all of these different types of query processing."

Flaw Two - Long Time to Solution, High Operating Costs
The second flaw relates to the narrower field of structured data which in and of itself presents significant challenges at enterprise scale.   Here Howard states, "I don't think the concept of a single monolithic EDW was ever the right one.  It was too time consuming and expensive to set up and run. But even if you think that it was conceptually the right approach it has been overtaken by practical considerations: in particular, data marts and application-specific appliances have proliferated throughout large enterprises and there is no way that that is going to change.  Further, data virtualisation and federation are now mature technologies that allow federated query access across data marts and warehouses so there is no longer any incentive to try and centralise everything."

Forrester Report on Data Virtualization Raises  Similar  EDW Concerns
Data Virtualization Reaches Critical Mass: Technology Advancements, New Patterns, And Customer Successes Make This Enterprise Technology Both A Short- And Long-Term Solution
by Brian Hopkins with Alex Cullen, Mike Gilpin, Boris Evelson, Gene Leganza, and Mackenzie Cahill of Forrester Research raises similar EDW concerns including:

  • Shortcomings in ETL-based data integration approaches cause degradation of data quality and delayed information.
  • Traditional data integration approaches that consolidate multiple databases into a single warehouse are too slow, expensive and risky in today's dynamic business environment.
  • Cloud-based data sources add new integration complexity
  • Business cannot wait for long running initiatives such as MDM and will take the integration problem into their own hands if IT cannot provide better short term alternatives.

Will Data Virtualization Be the New King?
The Forrester report predicts accelerated adoption over the next 18-36 months. Six technology advancements are accelerating data virtualization's ascension to the data management throne.  These include:

  • Cost-based query optimization performance improvements that open the door for additional use cases and wider adoption;
  • Distributed caching which sets the framework for global deployments;
  • Improved discovery tools that simplify understanding of data and accelerate new development;
  • Data masking and other security enhancements to better govern data delivery;
  • Third party technology integration such as CDC and ESB to extend data virtualization tooling; and
  • Big data integration to create new opportunities for analysis and insight.

Further, multiple Virtualization Journal articles highlight data virtualization's benefits including:

Is the Kingdom Large Enough for Two Kings?
One of the most important services that analyst firms such as Bloor and Forrester provide to their enterprise IT clients is the prediction of future trends.    But enterprises move more slowly and large EDW investments have and will continue to pay off for the foreseeable future.

Given this reality, maybe we should anticipate a long period of shared rule where data virtualization and data warehousing team to provide a benevolent reign.    If this comes to pass, take a look at these Virtualization Journal articles for opportunities to take advantage of this enlightened era.

More Stories By Robert Eve

Robert Eve is the EVP of Marketing at Composite Software, the data virtualization gold standard and co-author of Data Virtualization: Going Beyond Traditional Data Integration to Achieve Business Agility. Bob's experience includes executive level roles at leading enterprise software companies such as Mercury Interactive, PeopleSoft, and Oracle. Bob holds a Masters of Science from the Massachusetts Institute of Technology and a Bachelor of Science from the University of California at Berkeley.

IoT & Smart Cities Stories
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...
Cell networks have the advantage of long-range communications, reaching an estimated 90% of the world. But cell networks such as 2G, 3G and LTE consume lots of power and were designed for connecting people. They are not optimized for low- or battery-powered devices or for IoT applications with infrequently transmitted data. Cell IoT modules that support narrow-band IoT and 4G cell networks will enable cell connectivity, device management, and app enablement for low-power wide-area network IoT. B...
The hierarchical architecture that distributes "compute" within the network specially at the edge can enable new services by harnessing emerging technologies. But Edge-Compute comes at increased cost that needs to be managed and potentially augmented by creative architecture solutions as there will always a catching-up with the capacity demands. Processing power in smartphones has enhanced YoY and there is increasingly spare compute capacity that can be potentially pooled. Uber has successfully ...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...