Data Federation and the CMDB

This is the first in a series of posts looking into the fit between data federation and the Configuration Management Database (CMDB).  For those unfamiliar with CMDBs, here's a quick Wikipedia update:

A configuration management database (CMDB) is a repository of information related to all the components of an information system. Although repositories similar to CMDBs have been used by IT departments for many years, the term CMDB stems from ITIL. In the ITIL context, a CMDB represents the authorized configuration of the significant components of the IT environment. A CMDB helps an organization understand the relationships between these components and track their configuration.

What got me interested in data federation as it relates to CMDBs was the increasing frequency of the word "federation" in the marketing collateral and communications of several CMDB vendors (You don't have to call me twice to supper...).  It made perfect sense: with organizations so large and assets all over the globe, there seemed no way one could just have a database to store it all in.  Plus, it seemed to me a lot like reporting, which is a common use for data federation.

On my path to enlightenment, I came across Hank Marquis' post Enterprise CMDB, which helped me understand the core problem federation is aiming to solve.  Turns out it's not just that one needs to gather data from lots of distributed systems to create a federated CMDB, but it's the data model itself that cries out for federation:

But a CMDB does not store most of its data; it references data stored externally in other, perhaps relational, databases. And a CMDB is used to provide contextual awareness over non-obviously related bits of data. For example, a common inquiry posed to configuration management might be: "How many users, in sales, use SAP during the last week of the month?"

This kind of query is not well suited to a centralized relational database with pre-built SQL queries. There are just too many possible combinations of data. This type of query has to pull information from many systems, and the data it needs is probably not all nicely lined up in rows and columns ready to query. No, an enterprise CMDB has to be dimensional—a technology that represents data as different dimensions or plains.

And OLAP is not the answer:

Don't get too excited—simply having OLAP does not give you a CMDB for a couple of very special reasons. First, most CMDB data resides outside of the CMDB system. In order to pull this data from many sources requires federation—a new CMDB buzzword you will begin to hear about more and more.

By way of an example of federation and what it requires, lets consider an IT service for project management. Composing this service are human resource information residing in SAP, project management data in Microsoft Project Server, IT asset information in CA Unicenter, and networking hardware resource data discovered and stored in CiscoWorks.

There are, of course, challenges with things like reconciliation and synchronization.  That's where a product like Ipedo's comes in.  We already have may of these features built in to our system, providing great tools for someone trying to build a federated CMDB.

Here's how Hank puts it:

An enterprise CMDB is not a singular database, it is a complex system that has to federate other data stores, reconcile alternate views of the same data, detect unauthorized changes, synchronize approved changes with its own metadata store, and be able to dynamically represent configurations graphically on demand. This is no small task, and also the reason there are so few true CMDB solutions available today.

His last sentence really caught my attention.  Federation is a hard problem, which is why there are specialist vendors like us.  I think CMDB federation represents a huge upside for data federation

That's it for now.  I'll be digging into some more specifics in and around CMDB in future posts.

Fusing Together Oracle Products with Data and Service Virtualization

We did a demo recently that I thought was worth sharing.  With all the talk of Oracle Fusion in the press recently, I thought it was worth showing how simply and easily you can create custom combinations of data using Ipedo's data and service virtualization techniques.

In the demo we fuse together data from some Oracle 10g databases, with Salesforce.com (ex-Oracle CEO,  so we threw it in) and data moved from SQL Server by Oracle DI.  We put it all into Oracle BI Publisher, run Ipedo on Oracle AS, use another Oracle 10g database as a cache, and lastly move it into Oracle BPEL.  Whew!  Our professional services guys are now almost Oracle experts.

Here's the demo.  It's 15 minutes long, but broken into sections so you can navigate around quickly.

New Release of Ipedo's Virtual Integration Server

I'm very pleased to announce the new version of Ipedo XIP.  In 4.3, we've added a bunch of features to make the product even stronger.   I'm especially excited about the new incremental caching capabilities.

If you missed the announcement, here are the highlights:

The new enhancements are designed to increase both the performance and flexibility for large deployments, as well as accommodate the growing use of open source databases in the enterprise.  Whether creating data services in an SOA, or new reports in business intelligence applications, customers can access combinations of business information using a simple templated approach, with performance levels that are transparently maintained.  New features include:

 

-- Incremental Caching -- Ipedo XIP now allows for its cache to be
    refreshed incrementally, significantly reducing network traffic when a
    cache refresh is required.  Working automatically and in the background,
    the new incremental caching capability watches remote data sources for
    changes and automatically keeps the cache in sync.

-- Parameterized Views -- Parameterized views allow developers to
    encapsulate complex queries as parameterized templates, thereby avoiding
    construction of complex ad-hoc query strings at execution time. They are
    defined using a syntax common to prepared statements at design time, and
    parameter values are passed into the view at run time. Ipedo XIP performs
    all the optimizations on the resulting composite query tree to ensure
    efficient execution of the user query.

-- MySQL Enterprise Server Support -- Ipedo XIP can now work seamlessly
    with MySQL Enterprise Server, both as a data source and as a results cache.
    Extensive testing on data type interoperability and performance was done to
    ensure optimal deployment.

The new parameterized views are quite powerful.  MySQL support will be a hit in several of our accounts.  You can see the full release here. 

Data(base) Virtualization in the Mix

Came across a convenient summary of virtualization strategies today.  Though data and database virtualization have been discussed here and by analysts that cover integration, this is one of the first articles I've come across that discusses data and database virtualization as full fledged peers of the more familiar application and storage virtualization technologies.

A bit brief on the specifics, but a useful summary nonetheless.

Here's the link to the article.

Data Transformations in EII

From time to time I get asked about Ipedo's data transformation capabilities.  I was asked again a few weeks ago by a prospect, so I decided to put two of our developers to work.   I asked them to write a few posts that explain what Ipedo XIP can do in the area of data transformation.

Now we have two great entries that describe both halves of Ipedo's dual-core query engine: one on transforms using SQL, and one on transforms using XQuery.

Both of these just scratch the surface.  Just like the old Chinese expression "A long journey starts with the first step," these two entries only begin to expose the rich set of transformation capabilities available.  But they do cover some common cases in EII, which are often different from ETL needs. 

Let us know what you think.  If you want more examples, all you need to do is ask.  In the words of another Chinese proverb, "Dig a well before you are thirsty."

SOA, Virtual Machines and Virtual Databases

I was reading the latest post to Joe McKendrick's Service-Oriented Architecture this morning, 'IBM exec: no SOA-virtualization convergence, yet.'  It triggered something we've been kicking around here for some time - the idea of the virtual database.

Now, the IBM exec quoted in the blog was making the point that SOA and server virtualization - a la VMware - are not literally converging.  Well, sure, but is that really the point?  JavaScript and XML aren't literally converging, but together as AJAX they are powering a new breed of interactive Web Applications. 

The point really is in the motivation, the trend.  Joe's lead in sums it up pretty well:

"What's the common ground between SOA and virtualization? Both SOA and virtualization deal with abstraction of resources from across the network, and offer technology neutral interfaces removed from the original system."  [His emphasis]

Exactly.  And that's where we expect to see more and more attention being paid to database virtualization, aka EII.  Should anyone care that credit history is in one database, and contact info is in another, and recent purchases are in yet a third?  No.  All they want is the complete customer profile.  And if IT decides it wants to move from Oracle to MySQL, move the credit history table from SQL Server to Sybase, or run DB2 on VMware, who cares?  The nice thing about data virtualization is that it makes the data widely available to users and frees up IT to make optimal use of hardware, network bandwidth, and personnel.

With database virtualization, you can still send queries from an application or reporting tool, but the EII server is responsible for examining and routing all the queries - even caching the result locally if it expects 100 other users to want the same data (e.g. yesterday's sales).

Just as nice, if users decide they want the data as a Web Service, all the EII server admin has to do it publish the virtual view - customer profile, yesterday's sales - as a Web Service.  It takes about five minutes.

Todd Biske, whom Joe quotes in the entry, goes on to talk about how virtualization can be an effective tool for IT staff dealing with the complexity of SOA:

"If we don’t give the operational staff the tools they need to efficiently manage those resources, we’ll be in an even bigger mess."

We couldn't agree more, especially when it comes to data.  And no better tool exists for managing the complexity of databases in an SOA world than the database virtualization capabilities of EII. 

Calculating the ROI of EII: Application Maintenance (Example 5/5)

Because the EII layer can easily handle additions or changes to underlying data sources, organizations can focus on maintaining the EII layer and not the maintenance of every application and its connection. If a data source changes, the administrator only needs to update the mapping in the EII layer; all reports remain unchanged as they use the schema defined in the EII layer.

To figure out if this can save you money, think about this question: How much does it cost to maintain point-to-point connections among all your applications and data sources?

                                                                                         

	Example	Formula
Conventional   Approach
Cost to   develop a customized connector between two applications ($)	$10,000	A
Number of   connections required per year	25	B
Application   maintenance cost using conventional approach ($)	$250,000	A * B
Cost to change   a report because a data source changes ($)	$1,000	C
Number of   changes to reports required per year	200	D
Application   maintenance cost using conventional approach ($)	$200,000	C * D
EII   Approach
Cost to   develop a customized connector between two applications ($)	$0	E
Cost to change   a report because a data source changes ($)	$0	F
Cost to change   a data source mapping in the EII layer ($)	$100	G
Number of   changes to reports required per year	200	D
Application   maintenance cost using EII approach ($)	$20,000	G * D
Potential annual   cost savings from EII ($)	$430,000	(A * B) + (C *   D) - (G * D)

In this example, a company that eliminates customized programming for connecting applications and changing reports will save $430,000 by using EII.

• What value-add projects could your IT team complete if they are not changing report formats?
• How much more motivated would your developers be if they spent less time on software maintenance?

That concludes our five-part analysis.  Hopefully you've found some areas where there might be cost savings.  If you did and would like to take things further, here are some suggestions:

1. If you want to use your own numbers in the above examples, you can request the companion Excel spreadsheet from Ipedo.
2. If your business situation requires a customized financial model, one of our analysts can provide assistance with the process, or even build an interactive spreadsheet based on information that you provide.
3.  Once you complete your financial models and convince your team that EII looks good on paper, a logical next step is to develop a proof-of-concept application that demonstrates cost savings and/or productivity improvement in a controlled environment. Ipedo can help with this as part of the sales process.

Calculating the ROI of EII: Data Consolidation (Example 4/5)

Managing data costs a lot of money.  Just ask any CIO.  In any given company, there is a lot of money spent (and wasted) moving and copying data.  Enter EII.

Using EII to create a virtual, centralized database that cuts across disparate systems can reduce the cost of data maintenance, and eliminate the need for additional investment to store replicated data. Even with a conservative estimate of $50K per data mart ($25K hardware purchase and software licensing, $10K development cost, $5 software maintenance license, $10K personnel to manage the data mart), this gets expensive quickly.

The ROI-unlocking question: How much of your IT budget goes towards moving data around the enterprise and storing copies for different applications to use?

                             

Top-Down   Approach	Example	Formula
Annual IT budget   ($)	$10,000,000	A
% of resources   spent on data migration and replication (%)	25%	B
% improvement   with EII (%) (estimate)	10%	C
Potential   annual cost savings from EII ($)	$250,000	(A * B) * C

In this example, a company that spends 25% of its $10 million IT budget on data migration and data replication would save $250,000 (2.5% of overall IT budget) by using EII to complement its other data integration initiatives.

 

                       

Bottom-Up   Approach	Example	Formula
Cost to deploy   single new data mart ($)	$50,000	D
Number of data   marts deployed per year	10	E
Potential   annual cost savings from EII ($)	$500,000	D * E

In this example, a company that can eliminate 10 data marts per year by using EII will save $500,000.

And with all the focus on CDI and MDM these days, these upside benefits are sure to resonate:

• What is the value of having a “single version of the truth” based in the original data sources, rather than copies of data that become obsolete once the source changes
• How much time do you spend reconciling different versions of data residing in different locations?
• What politics are involved in getting permission to replicate data in your local data mart?
• How could you better deploy your IT resources if they did not have to manage a data mart?

That's it for today.  Next time, our final installment.

EII Haiku

In high technology, we often do things that have never been tried before.  I'm fairly certain this is one of them. 

Since I was recently accused of being too serious with my ROI posts, but not wanting to give way to the senseless prattling of many blogs, I offer you EII Haiku.  I like to think of them as lighthearted but deep.  Enjoy.

Queries federated
Distant rows and columns meet
My warehouse unscathed

Web Services, SOAP
Loosely coupled, lightly known
Join them, EII

Data is scattered
Metadata, SQL meet
All becomes as one

P.S. - If you would like to share your own, please feel free to do so in the Comments section.

Calculating the ROI of EII: Developer Productivity (Example 3/5)

With this entry, we'll move from the business side of the house to the development side of the house.  Yes, EII has something for everyone.

Recent IDC research reports that the average enterprise spends 40% of its IT budget on data integration (ETL, EII, EAI, etc.). Custom programming to connect disparate data sources is expensive to develop and maintain, especially in an environment of continual change. The unified abstraction layer that EII provides allows developers to write to one common interface to access multiple data sources, reducing the total development effort for new applications. The ability to write queries and not code allows departmental control of data integration, which saves the time and effort of going through a corporate IT group. These queries can be reused among different groups, further saving money.

So, What percentage of your IT budget do you spend on data integration tasks?   There are two ways to look at it.

                             

Top-Down Approach	Example	Formula
Annual IT budget ($)	$10,000,000	A
% spent on integration (%)   (ETL, EAI, EII, etc.)	40%	B
% improvement with EII (%)   (estimate)	10%	C
Potential annual cost savings   from EII ($)	$400,000	(A * B) * C

In this example, a company with a $10 million IT budget would save $400,000 annually (4% of overall IT budget) by including EII in their overall data integration strategy.

 

                                                           

Bottom-Up Approach	Example	Formula
Number of developers on   integration projects	20	A
Annual cost (including   benefits) of developers ($)	$120,000	B
% improvement with EII (%)   (estimate)	10%	C
Potential annual cost savings   from EII ($)	$240,000	(A * B) * C
Person-weeks for integration   projects over a year	1,000	D
Annual cost (including   benefits) of developers ($)	$120,000	E
% improvement with EII (%)   (estimate)	10%	F
Potential annual cost savings   from EII ($)	$240,000	(D * (E / 50)) * F

In these examples, a 20-person team would save $240,000 annually using EII as a part of their data integration projects.

And, as usual, some additional upside potential:

• What creative revenue generating or cost reducing functions will your team develop in the time they save by not rewriting software to accommodate changes in infrastructure or data format?
• How much more productive will your developers be if they focus on writing interesting applications rather than on more basic data integration tasks?

All for now.