Toward Simple Data Services

For those of you who might have missed it, we put out a new release of our flagship Ipedo XIP in December.  The theme was using EII in an SOA to create data services.

What we hear a lot when we're out talking to customers and prospects is that getting the data right in an SOA is hard, and that what's needed are tools to make it easier.   The last thing organizations want to do is write a lot of custom Java code just to get their SOA up and running.  Kind of defeats the purpose.

So what we've done is add a bunch of features to make it really simple and efficient to consume and publish Web Services.  You can see the full release for details, but here are the highlights:

• One-Click Publishing – The enhanced Web Services wizard allows simple one-click publishing of new Web Services, complete with security profiles, performance management, and automated WSDL generation.
  
• Dual Caching Option – Allows a Web Services View to be cached natively as XML, or in relational form, or both. Native caching improves performance and allows versioning of XML results. Using the relational option allows automatic decomposition of XML into relational form for use in other applications.
  
• Web Services Tables Metadata – Metadata enhancements for Ipedo's Web Services Tables display complete lineage and dependency information for inbound Web Services. Ipedo's Web Services Tables allow users to easily map XML-based Web Services into relational models.
  
• Join Hints for Web Services – Ipedo's query optimizer can now accept hints, including join strategy and join order, that minimize the number of Web Services calls needed to fulfill a query request, and thereby improve overall performance.
  
• XQuery for SOA – Ipedo's XQuery engine has been tuned to provide greater insight and better performance for SOA with a range of advanced functions, including XML payload introspection, fast transformation, content validation, and service metadata introspection. In addition, XQuery can be used to trigger subsequent 'cascading queries' based on the payload content of a Web Service.

I'd also like to put in a plug for our long XQuery heritage.  I think people are finally starting to realize how hard it is to deal with all the data in SOA payloads, and how good a tool XQuery is for the job.

Linthicum on the Importance of Data

Came across a nice podcast by David Linthicum today.  As you know from reading this blog, we're big believers in the importance of data in SOA - what we call data services.  In fact, we often wonder how people get their SOA projects off the ground without giving this serious consideration.

In his podcast, David talks about this very point.  Nice to hear an SOA expert using the terms 'data services' and 'virtual database.'  Take a listen.

You might also be interested in a new entry from our technical folks called The One-click Web Service.  To me it illustrates how powerful an EII server can be in an SOA project: all the benefits of SOA, simplicity of deployment, and high-performance data access.

SOA, Virtual Machines and Virtual Databases

I was reading the latest post to Joe McKendrick's Service-Oriented Architecture this morning, 'IBM exec: no SOA-virtualization convergence, yet.'  It triggered something we've been kicking around here for some time - the idea of the virtual database.

Now, the IBM exec quoted in the blog was making the point that SOA and server virtualization - a la VMware - are not literally converging.  Well, sure, but is that really the point?  JavaScript and XML aren't literally converging, but together as AJAX they are powering a new breed of interactive Web Applications. 

The point really is in the motivation, the trend.  Joe's lead in sums it up pretty well:

"What's the common ground between SOA and virtualization? Both SOA and virtualization deal with abstraction of resources from across the network, and offer technology neutral interfaces removed from the original system."  [His emphasis]

Exactly.  And that's where we expect to see more and more attention being paid to database virtualization, aka EII.  Should anyone care that credit history is in one database, and contact info is in another, and recent purchases are in yet a third?  No.  All they want is the complete customer profile.  And if IT decides it wants to move from Oracle to MySQL, move the credit history table from SQL Server to Sybase, or run DB2 on VMware, who cares?  The nice thing about data virtualization is that it makes the data widely available to users and frees up IT to make optimal use of hardware, network bandwidth, and personnel.

With database virtualization, you can still send queries from an application or reporting tool, but the EII server is responsible for examining and routing all the queries - even caching the result locally if it expects 100 other users to want the same data (e.g. yesterday's sales).

Just as nice, if users decide they want the data as a Web Service, all the EII server admin has to do it publish the virtual view - customer profile, yesterday's sales - as a Web Service.  It takes about five minutes.

Todd Biske, whom Joe quotes in the entry, goes on to talk about how virtualization can be an effective tool for IT staff dealing with the complexity of SOA:

"If we don’t give the operational staff the tools they need to efficiently manage those resources, we’ll be in an even bigger mess."

We couldn't agree more, especially when it comes to data.  And no better tool exists for managing the complexity of databases in an SOA world than the database virtualization capabilities of EII. 

Calculating the ROI of EII: Application Maintenance (Example 5/5)

Because the EII layer can easily handle additions or changes to underlying data sources, organizations can focus on maintaining the EII layer and not the maintenance of every application and its connection. If a data source changes, the administrator only needs to update the mapping in the EII layer; all reports remain unchanged as they use the schema defined in the EII layer.

To figure out if this can save you money, think about this question: How much does it cost to maintain point-to-point connections among all your applications and data sources?

                                                                                         

	Example	Formula
Conventional   Approach
Cost to   develop a customized connector between two applications ($)	$10,000	A
Number of   connections required per year	25	B
Application   maintenance cost using conventional approach ($)	$250,000	A * B
Cost to change   a report because a data source changes ($)	$1,000	C
Number of   changes to reports required per year	200	D
Application   maintenance cost using conventional approach ($)	$200,000	C * D
EII   Approach
Cost to   develop a customized connector between two applications ($)	$0	E
Cost to change   a report because a data source changes ($)	$0	F
Cost to change   a data source mapping in the EII layer ($)	$100	G
Number of   changes to reports required per year	200	D
Application   maintenance cost using EII approach ($)	$20,000	G * D
Potential annual   cost savings from EII ($)	$430,000	(A * B) + (C *   D) - (G * D)

In this example, a company that eliminates customized programming for connecting applications and changing reports will save $430,000 by using EII.

• What value-add projects could your IT team complete if they are not changing report formats?
• How much more motivated would your developers be if they spent less time on software maintenance?

That concludes our five-part analysis.  Hopefully you've found some areas where there might be cost savings.  If you did and would like to take things further, here are some suggestions:

1. If you want to use your own numbers in the above examples, you can request the companion Excel spreadsheet from Ipedo.
2. If your business situation requires a customized financial model, one of our analysts can provide assistance with the process, or even build an interactive spreadsheet based on information that you provide.
3.  Once you complete your financial models and convince your team that EII looks good on paper, a logical next step is to develop a proof-of-concept application that demonstrates cost savings and/or productivity improvement in a controlled environment. Ipedo can help with this as part of the sales process.

Calculating the ROI of EII: Data Consolidation (Example 4/5)

Managing data costs a lot of money.  Just ask any CIO.  In any given company, there is a lot of money spent (and wasted) moving and copying data.  Enter EII.

Using EII to create a virtual, centralized database that cuts across disparate systems can reduce the cost of data maintenance, and eliminate the need for additional investment to store replicated data. Even with a conservative estimate of $50K per data mart ($25K hardware purchase and software licensing, $10K development cost, $5 software maintenance license, $10K personnel to manage the data mart), this gets expensive quickly.

The ROI-unlocking question: How much of your IT budget goes towards moving data around the enterprise and storing copies for different applications to use?

                             

Top-Down   Approach	Example	Formula
Annual IT budget   ($)	$10,000,000	A
% of resources   spent on data migration and replication (%)	25%	B
% improvement   with EII (%) (estimate)	10%	C
Potential   annual cost savings from EII ($)	$250,000	(A * B) * C

In this example, a company that spends 25% of its $10 million IT budget on data migration and data replication would save $250,000 (2.5% of overall IT budget) by using EII to complement its other data integration initiatives.

 

                       

Bottom-Up   Approach	Example	Formula
Cost to deploy   single new data mart ($)	$50,000	D
Number of data   marts deployed per year	10	E
Potential   annual cost savings from EII ($)	$500,000	D * E

In this example, a company that can eliminate 10 data marts per year by using EII will save $500,000.

And with all the focus on CDI and MDM these days, these upside benefits are sure to resonate:

• What is the value of having a “single version of the truth” based in the original data sources, rather than copies of data that become obsolete once the source changes
• How much time do you spend reconciling different versions of data residing in different locations?
• What politics are involved in getting permission to replicate data in your local data mart?
• How could you better deploy your IT resources if they did not have to manage a data mart?

That's it for today.  Next time, our final installment.

EII Haiku

In high technology, we often do things that have never been tried before.  I'm fairly certain this is one of them. 

Since I was recently accused of being too serious with my ROI posts, but not wanting to give way to the senseless prattling of many blogs, I offer you EII Haiku.  I like to think of them as lighthearted but deep.  Enjoy.

Queries federated
Distant rows and columns meet
My warehouse unscathed

Web Services, SOAP
Loosely coupled, lightly known
Join them, EII

Data is scattered
Metadata, SQL meet
All becomes as one

P.S. - If you would like to share your own, please feel free to do so in the Comments section.

Calculating the ROI of EII: Developer Productivity (Example 3/5)

With this entry, we'll move from the business side of the house to the development side of the house.  Yes, EII has something for everyone.

Recent IDC research reports that the average enterprise spends 40% of its IT budget on data integration (ETL, EII, EAI, etc.). Custom programming to connect disparate data sources is expensive to develop and maintain, especially in an environment of continual change. The unified abstraction layer that EII provides allows developers to write to one common interface to access multiple data sources, reducing the total development effort for new applications. The ability to write queries and not code allows departmental control of data integration, which saves the time and effort of going through a corporate IT group. These queries can be reused among different groups, further saving money.

So, What percentage of your IT budget do you spend on data integration tasks?   There are two ways to look at it.

                             

Top-Down Approach	Example	Formula
Annual IT budget ($)	$10,000,000	A
% spent on integration (%)   (ETL, EAI, EII, etc.)	40%	B
% improvement with EII (%)   (estimate)	10%	C
Potential annual cost savings   from EII ($)	$400,000	(A * B) * C

In this example, a company with a $10 million IT budget would save $400,000 annually (4% of overall IT budget) by including EII in their overall data integration strategy.

 

                                                           

Bottom-Up Approach	Example	Formula
Number of developers on   integration projects	20	A
Annual cost (including   benefits) of developers ($)	$120,000	B
% improvement with EII (%)   (estimate)	10%	C
Potential annual cost savings   from EII ($)	$240,000	(A * B) * C
Person-weeks for integration   projects over a year	1,000	D
Annual cost (including   benefits) of developers ($)	$120,000	E
% improvement with EII (%)   (estimate)	10%	F
Potential annual cost savings   from EII ($)	$240,000	(D * (E / 50)) * F

In these examples, a 20-person team would save $240,000 annually using EII as a part of their data integration projects.

And, as usual, some additional upside potential:

• What creative revenue generating or cost reducing functions will your team develop in the time they save by not rewriting software to accommodate changes in infrastructure or data format?
• How much more productive will your developers be if they focus on writing interesting applications rather than on more basic data integration tasks?

All for now.

Calculating the ROI of EII: Analyst Utilization (Example 2/5)

Here's one of my favorites, inspired by a conversation we had with a prospect back in 2005.  It's a great product management moment when the benefits of your product dovetail so nicely with a problem a customer has.

To wit, a vice president at a commercial bank that did not use EII told us that the analysts there spend about 60% of their time on data collection and only 40% on analysis. Moving from a 40%-60% split to a 90%-10% split means more than doubling (125%) individual analyst productivity. Thus, one analyst can do the work of slightly more than two. EII allows analysts to simplify the data collection process and spend more time producing useful, actionable information.

So, here's the question to unlock your potential ROI: How much time do your analysts spend searching for data?

                                   

	Example	Formula
Percentage of   time analysts spend on value-added activities pre-EII (%)	40%	A
Percentage of   time analysts spend on value-added activities post-EII (%)	90%	B
Number of   analysts on your team	10	C
Annual cost   (including benefits) of analyst ($)	$100,000	D
Potential annual   cost savings from EII ($)	$555,556	((B - A) / B)   * C * D

In this example, a team of 10 analysts does about 20 person-days of analysis per week pre-EII and 45 person-days per week post-EII, for a 125% improvement. If the amount of work was fixed at 20 person-days per week, a team of 5 analysts (actually 4.44) could do the job that previously required 10.

Additional Upside

Still need more ammo?  Think about quantifying these:

· What new discoveries will your analysts make when they have more time to focus on analysis?

· How many errors will be eliminated by reducing manual handling of data and what will the cost savings be of eliminating these errors?

· By spending more time using data rather than looking for it, how much more valuable will the analysts become to the organization?

· What is the effect on job satisfaction if people are doing what they enjoy, rather than just collecting data?

That's it for this installment.  Three more to go.  Stay tuned.  If you can't wait, check out the complete ROI guide, which can be found in Ipedo's EII Zone.

"Data Driving SOA." Amen, Brother.

I was pleased to see Ronan Bradley's post from the other day 'Data: The Heart of SOA.'  I liked it so much I almost stole the title.  In it, he talks about how data issues really are part and parcel of any SOA project.  Amen, brother.

Once SOA projects get beyond simple 'Hello world' examples, data becomes critical.  First, you need something to pump into the service.  Then, of course, you need to make sure the data is correct.  So while much of the talk has been on service-enabling applications, this is all meaningless unless you can service-enable the data.  People in the data integration space like to refer to these as 'Data Web Services' as opposed to 'Application Web Services,' and often just 'Data Services' for short.

The keys here are data models and data access.  Data models allow you to know what the data looks like, what it means, and who owns it.  Data access is just what it says it is, but involves performance and security (and maybe even governance) issues.

I recently wrote about how EII is perfect for creating data services from existing sources - it provides the mediation layer to address performance, security and governance concerns.  To Ronan's point, Ipedo's product also allows the dynamic creation of data services from existing data models:

What's even cooler with Ipedo's product is that all of this can be scripted.  As my colleague Peter Spielvogel wrote about in The EII Files, Ipedo can automatically import data models from popular modeling tools like ERwin and ER/Studio.  So you can import all of the data models you need in a matter of minutes.  Once in Ipedo, you can script which Views you would like to be made available as Web Services, and voila!

I think we'll see more and more companies looking for ways to leverage their existing data models, which are typically relational, and transform them to Data Services. 

Dave Linthicum in his post "'SOA is Data' Really?" takes issue with Ronan's emphasis on data over process.  While I don't dispute that process and functions are the focus of an SOA application, a few things seem clear.  First, data services can support application services (get me customer data, get me risk profile, get me consolidated sales figures, etc.).  Second, many Web Services really are just data services, allowing easy access to information.  Third, seems to me that RESTful Web Services, being gets and puts, are predominantly Data Services by their nature.

That said, I also think you'll see more and more use of EII within ESBs, where ESBs will call a Data Service that is mediated by an EII product.  EII servers are just better suited to this task.  So the process- or application-oriented bus products can do what they do best.

Can I get an Amen?

EII and the Bus

A picture is worth a thousand words, as they say.  Based on all the words that have been written about ESB, I'd say the picture below is worth even more.

Bill Inmon had the following picture of why a bus is not an integration panacea is his recent B-Eye Network article The Bus as a Substitute for Integration, Part I:

Now you could read the article, but this picture sums it up, doesn't it?  That's how it struck me.  Creating a powerful, yet simple, picture of a complex issue is hard, and takes a foundation of expertise in a topic area, and still then some sweating over. (For some great examples of excellence in infographics, I highly recommend The Visual Display of Quantitative Information, with my favorite Napoleon's March chart.)

Bill Inmon certainly knows a thing or two about data integration, which is perhaps why this picture is so poignant.  And while it is not about EII per se, it reminded me of many conversations I've had with ESB proponents over the past year.  The point I always try to impress on people is that ESBs are good at reliably moving data, and they are (thankfully) more open than some of the older bus technologies, but they still don't help you integrate the data.  The reply to which - and I always love this one - is "All you need to do is write some code..."  If I had a dollar for every time I heard that one...

Here's a better alternative.  Create a parameterized View of what you want in an EII product like Ipedo's.  A View is essentially a virtual representation of the data you want, joined, cleaned, and - importantly - semantically correct.  Then, instead of the typical bus flow of Get A [invoke custom code] then Get B [invoke custom code] then Get C [invoke custom code] and so on, the View queries multiple data sources and returns the information your application needs to the bus.  It's not magic, just a cleaner, easier approach with a more suitable tool.

So before you jump on the bus bandwagon, take another look at that picture and think about how EII can help.