An EII/ETL Détente – Can’t We All Just Get Along?

There seems to be quite a bit of angst out there among the data integration crowd. So much wrangling over the new kid on the block, EII.

Wars of words often start with differences of ideology. When EII arrived, some vendors (wrongly) suggested that EII was the new data warehouse, replacing the old data warehouse and ETL process with a new, virtual data warehouse. This affront was met with a torrent of criticisms against EII. Articles with very angry-sounding titles appeared. Most were, predictably, written by ETL vendors looking to excoriate EII.

Recently, the ice in this cold war has started to thaw. In the past nine months, a new wave of more enlightened articles has emerged. First Neil Raden surveyed the integration landscape in December’s Intelligent Enterprise. Greg Mancuso and Al Moreno from Sinecon wrote an excellent overview of available integration strategies in April’s DM Review. Curt Hall from the Cutter Consortium wrote an excellent piece on the fit between EII and Data Warehousing in May. And hopefully that’s just the beginning.

Speaking for my fellow EII vendors, I can say that we hold certain truths to be self-evident: that there is no better way to load a data warehouse than ETL; that ETL performance and transformation capabilities are better suited for large volume data transfers; that we have no designs on replacing data warehouses.

Rather, we see an exciting future for EII. It will be a piece of the integration landscape, working together alongside ETL. We see innovative ways of delivering real-time information. We see innovative ways of delivering data from data warehouses. There are kinks to be worked out, to be sure, but this is true of every new technology.

Based on customer reaction, EII has found its place and will continue to grow. Over the past few years, significant improvements in the areas of performance and metadata integration have been made. Advances in transformation and caching capabilities continue to be made.

 To borrow and repurpose a quote idea from Neil’s article, as Victor Hugo once said "There is nothing more powerful than an idea whose time has come." We think the time is now for EII. Rather than fighting a war of words, cooperation is in order. Only a détente will ultimately bring the right balance of technologies to customers. So can’t we all just get along?

EII, Peak Performance, and BPM

Another guest post this week by Peter Spielvogel, resident cycling and EII maven. -T.M.

By Peter Spielvogel

Running a business has a number of similarities with running a Tour de France team (although I have no direct experience with the latter). In both cases, teams that use data better than their competitors gain a significant competitive advantage. And, the volume of data and how it’s exploited have changed dramatically over the past few years.

The BPM in the title refers to the beats per minute that riders carefully monitor to make sure they maintain a pace they can sustain. During the preparation phase for the Tour, each rider’s miles are logged, along with power output, heart rate, speed, cadence, for each ride. All this information, plus data on diet and rest is stored in a database and compared to prior performance metrics and benchmark data. Trainers, riders, and managers recommend changes, test them in a controlled environment, and monitor the results. New technologies, such as wireless, enable real-time evaluation and tuning of performance, all while on the move.

Does any of this sound like your day-to-day activities in corporate IT? Over the past few years, the amount of corporate data has exploded. Determining what’s important and how to extract value from a bunch of numbers becomes increasingly difficult. Fortunately, data integration technologies such as enterprise information integration (EII) have made this much easier. EII allows organizations to extract the information they need, on-demand, from wherever this information resides. This technology complements business intelligence (BI) tools and business performance management (BPM) software that enables complex multidimensional analysis.

Having real-time information at your fingertips allows you to answer difficult questions quickly. For example, when a competitor attacks, you need to be able to know how much you have in reserve so you can determine how to meet the challenge. Do you try to stay with the leader, hang back and hope they will burn out, or do your best to maintain position while saving energy for another day? Without precise information on max heart rate, VO2 max, power output, and the road ahead, making that call would be an educated guess based on gut-feel. Good managers are forced to make these kinds of decisions daily, but in today’s business environment, intuition is sometimes not enough. Unlike the Tour, business campaigns last more than three weeks and the market dynamics are more complex.

So, when you see riders with wires coming off their bodies and bikes, think about the flow of data and how technology has changed the rules of the game. Those that deploy new IT effectively can keep their BPM within the zone. Those who don’t will ultimately lose the race.

May the best team and the best technology win.

What EII Brings to BI and Reporting

     The key to making informed decisions is having the right data in the right place at the right time.  Executives and line managers rely on business intelligence (BI) and reporting tools to deliver timely, accurate, and relevant data for both strategic as well as operational decisions.  Simple, right?

     Not so simple.  Problem is, each party in the corporate information supply chain has different requirements regarding BI and reporting tools.

• Users of enterprise reports and BI need information in the proper format, regardless of the origin of the data or how many sources contain the necessary components.
  
  
• Application developers and system architects need a way to reduce the time they spend making custom reports.
  
  
• IT managers need a way to control access to back end systems, safeguard data, and reduce the amount of data replication used to feed BI tools.

    EII provides an information integration layer that complements BI and enterprise reporting tools, whether Business Objects, Hyperion, Cognos, Crystal Reports, Microsoft Reporting Services, or just Excel.  This layer can solve the various issues different groups have.

Here's a quick rundown of where and how EII can help:

Extends access to real-time information
EII uses a query-based approach to provide information on demand to BI software. This guarantees current data. For less time-sensitive applications, the EII layer can cache frequently used queries to reduce load on back-end systems.

Provides access to data from multiple relational sources
EII makes all data universally accessible by consolidating different data topologies, formats, and locations.  The EII layer appears to BI tools as a single source - containing fresh information.

Incorporates documents, messages, and other unstructured data
EII hides the complexity of multiple databases, content repositories, and message streams, each with its own different access protocols. EII can provide a flexible and scalable infrastructure that makes BI tools more powerful, useful, and valuable.

Reduces data replication
EII eliminates the need to replicate local data marts just to create reports. In addition, the EII  layer's capabilities reduce the load on production systems by retrieving only the required information and making it reusable to multiple tools rather than having each one perform its own queries.

     Even SOA is getting into the act.  Phil Wainright writes about how EII helps BI adapt to SOA in a recent Loosely Coupled post.

The Long Tail of Data

This week I turn over the keyboard to my colleague Peter Spielvogel so I can take a long needed summer vacation.  I asked him to write down something he mentioned to me a few weeks ago that I thought was pretty interesting. - T.M.

by Peter Spielvogel

     An article in the business section of today’s San Jose Mercury News on the concept of “The Long Tail” (free, but registration required) made me think about the latent value in the long tail of corporate data. In summary, the “long tail," a term popularized in an article in Wired Magazine, refers to the right section of a steeply declining curve that approaches the X-axis. The premise is that there is money to be made in selling items that are not necessarily best sellers, assuming the carrying costs are not too high. In one example the article cited, Apple’s iTunes site carries 50 times more titles than Wal-Mart. In another, sales of books on Amazon.com that are not even sold in a typical book store comprise the majority of revenue.

     How does this pertain to enterprise information? Certainly, much data resides in relational databases that people and applications access frequently. This "best selling data" is the analog to the top selling books or CDs mentioned in the article, and is likely "popular" because the data is relatively straightforward and easy to query using SQL.

     But, what about the data in the long tail? If ways were available to reach into the other corporate data beyond the reach of SQL, would that increase its use? Unstructured or semi-structured data sitting on isolated servers or individual PCs could contain valuable information about trends or history that might hold the answers to interesting questions. New technologies such as XQuery provide the ability to reach into these areas and allow people and applications to use this data to make decisions.

     Where will Web Services fall in the data graph - on the left among the best sellers or off in the long tail? Or, will Web Services sort themselves out into the popular (stock quotes, interest rates, weather, zip code finder) and the obscure (e.g. lotto number pickers, translate “hello” into different languages).

 

     Data integration solutions such as EII give companies a low-overhead way to extract value from all their enterprise data. They can query any source to get the information they need, without moving or replicating the data. With access to all their data, corporate decision makers have access to information with the breadth of iTunes at Wal-Mart prices.