Teasing Out 'Integration 2.0'

Those of you who have read my stuff know that this '2.0' bug has been knocking around my brain recently.  It's not so much a cheap desire to gather more buzz for Ipedo (although, by way of disclaimer, I should point out that this is what I am paid to do for my day job), but rather something that keeps nagging me to ask if there's not something to it.

Tim O'Reilly is most famous for declaring Web 2.0 in his article What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software.   His article tried to define what was a very elusive and potentially out of control concept, on the way to becoming a meaningless buzzword.  Along the way he discovered a set of design patterns that were undeniable.  Not that everything was totally new - there were people doing 2.0 stuff in the 1.0 era - but overall there was a pattern.  I liked the following way of describing Web 2.0:

You can visualize Web 2.0 as a set of principles and practices that tie together a veritable solar system of sites that demonstrate some or all of those principles, at a varying distance from that core.

We are on the verge of 2.0 everything.  I see many of these as hollow marketing efforts.  On the other hand, I see 2.0 as a challenge - a new way of thinking.   Our new 'ng' or '.next.' 

So far, the integration vendors have not really done anything to define what the next generation of integration should look like.  Saying that it's SOA is a punt.  That's not an answer.  The question is whether there is something new, something that can be hosted, a service, collaborative, open that will cost less and yield more.

Stay tuned here as we ponder.

SOA, Virtual Machines and Virtual Databases

I was reading the latest post to Joe McKendrick's Service-Oriented Architecture this morning, 'IBM exec: no SOA-virtualization convergence, yet.'  It triggered something we've been kicking around here for some time - the idea of the virtual database.

Now, the IBM exec quoted in the blog was making the point that SOA and server virtualization - a la VMware - are not literally converging.  Well, sure, but is that really the point?  JavaScript and XML aren't literally converging, but together as AJAX they are powering a new breed of interactive Web Applications. 

The point really is in the motivation, the trend.  Joe's lead in sums it up pretty well:

"What's the common ground between SOA and virtualization? Both SOA and virtualization deal with abstraction of resources from across the network, and offer technology neutral interfaces removed from the original system."  [His emphasis]

Exactly.  And that's where we expect to see more and more attention being paid to database virtualization, aka EII.  Should anyone care that credit history is in one database, and contact info is in another, and recent purchases are in yet a third?  No.  All they want is the complete customer profile.  And if IT decides it wants to move from Oracle to MySQL, move the credit history table from SQL Server to Sybase, or run DB2 on VMware, who cares?  The nice thing about data virtualization is that it makes the data widely available to users and frees up IT to make optimal use of hardware, network bandwidth, and personnel.

With database virtualization, you can still send queries from an application or reporting tool, but the EII server is responsible for examining and routing all the queries - even caching the result locally if it expects 100 other users to want the same data (e.g. yesterday's sales).

Just as nice, if users decide they want the data as a Web Service, all the EII server admin has to do it publish the virtual view - customer profile, yesterday's sales - as a Web Service.  It takes about five minutes.

Todd Biske, whom Joe quotes in the entry, goes on to talk about how virtualization can be an effective tool for IT staff dealing with the complexity of SOA:

"If we don’t give the operational staff the tools they need to efficiently manage those resources, we’ll be in an even bigger mess."

We couldn't agree more, especially when it comes to data.  And no better tool exists for managing the complexity of databases in an SOA world than the database virtualization capabilities of EII. 

Calculating the ROI of EII: Application Maintenance (Example 5/5)

Because the EII layer can easily handle additions or changes to underlying data sources, organizations can focus on maintaining the EII layer and not the maintenance of every application and its connection. If a data source changes, the administrator only needs to update the mapping in the EII layer; all reports remain unchanged as they use the schema defined in the EII layer.

To figure out if this can save you money, think about this question: How much does it cost to maintain point-to-point connections among all your applications and data sources?

                                                                                         

	Example	Formula
Conventional   Approach
Cost to   develop a customized connector between two applications ($)	$10,000	A
Number of   connections required per year	25	B
Application   maintenance cost using conventional approach ($)	$250,000	A * B
Cost to change   a report because a data source changes ($)	$1,000	C
Number of   changes to reports required per year	200	D
Application   maintenance cost using conventional approach ($)	$200,000	C * D
EII   Approach
Cost to   develop a customized connector between two applications ($)	$0	E
Cost to change   a report because a data source changes ($)	$0	F
Cost to change   a data source mapping in the EII layer ($)	$100	G
Number of   changes to reports required per year	200	D
Application   maintenance cost using EII approach ($)	$20,000	G * D
Potential annual   cost savings from EII ($)	$430,000	(A * B) + (C *   D) - (G * D)

In this example, a company that eliminates customized programming for connecting applications and changing reports will save $430,000 by using EII.

• What value-add projects could your IT team complete if they are not changing report formats?
• How much more motivated would your developers be if they spent less time on software maintenance?

That concludes our five-part analysis.  Hopefully you've found some areas where there might be cost savings.  If you did and would like to take things further, here are some suggestions:

1. If you want to use your own numbers in the above examples, you can request the companion Excel spreadsheet from Ipedo.
2. If your business situation requires a customized financial model, one of our analysts can provide assistance with the process, or even build an interactive spreadsheet based on information that you provide.
3.  Once you complete your financial models and convince your team that EII looks good on paper, a logical next step is to develop a proof-of-concept application that demonstrates cost savings and/or productivity improvement in a controlled environment. Ipedo can help with this as part of the sales process.

Calculating the ROI of EII: Data Consolidation (Example 4/5)

Managing data costs a lot of money.  Just ask any CIO.  In any given company, there is a lot of money spent (and wasted) moving and copying data.  Enter EII.

Using EII to create a virtual, centralized database that cuts across disparate systems can reduce the cost of data maintenance, and eliminate the need for additional investment to store replicated data. Even with a conservative estimate of $50K per data mart ($25K hardware purchase and software licensing, $10K development cost, $5 software maintenance license, $10K personnel to manage the data mart), this gets expensive quickly.

The ROI-unlocking question: How much of your IT budget goes towards moving data around the enterprise and storing copies for different applications to use?

                             

Top-Down   Approach	Example	Formula
Annual IT budget   ($)	$10,000,000	A
% of resources   spent on data migration and replication (%)	25%	B
% improvement   with EII (%) (estimate)	10%	C
Potential   annual cost savings from EII ($)	$250,000	(A * B) * C

In this example, a company that spends 25% of its $10 million IT budget on data migration and data replication would save $250,000 (2.5% of overall IT budget) by using EII to complement its other data integration initiatives.

 

                       

Bottom-Up   Approach	Example	Formula
Cost to deploy   single new data mart ($)	$50,000	D
Number of data   marts deployed per year	10	E
Potential   annual cost savings from EII ($)	$500,000	D * E

In this example, a company that can eliminate 10 data marts per year by using EII will save $500,000.

And with all the focus on CDI and MDM these days, these upside benefits are sure to resonate:

• What is the value of having a “single version of the truth” based in the original data sources, rather than copies of data that become obsolete once the source changes
• How much time do you spend reconciling different versions of data residing in different locations?
• What politics are involved in getting permission to replicate data in your local data mart?
• How could you better deploy your IT resources if they did not have to manage a data mart?

That's it for today.  Next time, our final installment.

EII Haiku

In high technology, we often do things that have never been tried before.  I'm fairly certain this is one of them. 

Since I was recently accused of being too serious with my ROI posts, but not wanting to give way to the senseless prattling of many blogs, I offer you EII Haiku.  I like to think of them as lighthearted but deep.  Enjoy.

Queries federated
Distant rows and columns meet
My warehouse unscathed

Web Services, SOAP
Loosely coupled, lightly known
Join them, EII

Data is scattered
Metadata, SQL meet
All becomes as one

P.S. - If you would like to share your own, please feel free to do so in the Comments section.

Calculating the ROI of EII: Developer Productivity (Example 3/5)

With this entry, we'll move from the business side of the house to the development side of the house.  Yes, EII has something for everyone.

Recent IDC research reports that the average enterprise spends 40% of its IT budget on data integration (ETL, EII, EAI, etc.). Custom programming to connect disparate data sources is expensive to develop and maintain, especially in an environment of continual change. The unified abstraction layer that EII provides allows developers to write to one common interface to access multiple data sources, reducing the total development effort for new applications. The ability to write queries and not code allows departmental control of data integration, which saves the time and effort of going through a corporate IT group. These queries can be reused among different groups, further saving money.

So, What percentage of your IT budget do you spend on data integration tasks?   There are two ways to look at it.

                             

Top-Down Approach	Example	Formula
Annual IT budget ($)	$10,000,000	A
% spent on integration (%)   (ETL, EAI, EII, etc.)	40%	B
% improvement with EII (%)   (estimate)	10%	C
Potential annual cost savings   from EII ($)	$400,000	(A * B) * C

In this example, a company with a $10 million IT budget would save $400,000 annually (4% of overall IT budget) by including EII in their overall data integration strategy.

 

                                                           

Bottom-Up Approach	Example	Formula
Number of developers on   integration projects	20	A
Annual cost (including   benefits) of developers ($)	$120,000	B
% improvement with EII (%)   (estimate)	10%	C
Potential annual cost savings   from EII ($)	$240,000	(A * B) * C
Person-weeks for integration   projects over a year	1,000	D
Annual cost (including   benefits) of developers ($)	$120,000	E
% improvement with EII (%)   (estimate)	10%	F
Potential annual cost savings   from EII ($)	$240,000	(D * (E / 50)) * F

In these examples, a 20-person team would save $240,000 annually using EII as a part of their data integration projects.

And, as usual, some additional upside potential:

• What creative revenue generating or cost reducing functions will your team develop in the time they save by not rewriting software to accommodate changes in infrastructure or data format?
• How much more productive will your developers be if they focus on writing interesting applications rather than on more basic data integration tasks?

All for now.

"Data Driving SOA." Amen, Brother.

I was pleased to see Ronan Bradley's post from the other day 'Data: The Heart of SOA.'  I liked it so much I almost stole the title.  In it, he talks about how data issues really are part and parcel of any SOA project.  Amen, brother.

Once SOA projects get beyond simple 'Hello world' examples, data becomes critical.  First, you need something to pump into the service.  Then, of course, you need to make sure the data is correct.  So while much of the talk has been on service-enabling applications, this is all meaningless unless you can service-enable the data.  People in the data integration space like to refer to these as 'Data Web Services' as opposed to 'Application Web Services,' and often just 'Data Services' for short.

The keys here are data models and data access.  Data models allow you to know what the data looks like, what it means, and who owns it.  Data access is just what it says it is, but involves performance and security (and maybe even governance) issues.

I recently wrote about how EII is perfect for creating data services from existing sources - it provides the mediation layer to address performance, security and governance concerns.  To Ronan's point, Ipedo's product also allows the dynamic creation of data services from existing data models:

What's even cooler with Ipedo's product is that all of this can be scripted.  As my colleague Peter Spielvogel wrote about in The EII Files, Ipedo can automatically import data models from popular modeling tools like ERwin and ER/Studio.  So you can import all of the data models you need in a matter of minutes.  Once in Ipedo, you can script which Views you would like to be made available as Web Services, and voila!

I think we'll see more and more companies looking for ways to leverage their existing data models, which are typically relational, and transform them to Data Services. 

Dave Linthicum in his post "'SOA is Data' Really?" takes issue with Ronan's emphasis on data over process.  While I don't dispute that process and functions are the focus of an SOA application, a few things seem clear.  First, data services can support application services (get me customer data, get me risk profile, get me consolidated sales figures, etc.).  Second, many Web Services really are just data services, allowing easy access to information.  Third, seems to me that RESTful Web Services, being gets and puts, are predominantly Data Services by their nature.

That said, I also think you'll see more and more use of EII within ESBs, where ESBs will call a Data Service that is mediated by an EII product.  EII servers are just better suited to this task.  So the process- or application-oriented bus products can do what they do best.

Can I get an Amen?

EII and the Bus

A picture is worth a thousand words, as they say.  Based on all the words that have been written about ESB, I'd say the picture below is worth even more.

Bill Inmon had the following picture of why a bus is not an integration panacea is his recent B-Eye Network article The Bus as a Substitute for Integration, Part I:

Now you could read the article, but this picture sums it up, doesn't it?  That's how it struck me.  Creating a powerful, yet simple, picture of a complex issue is hard, and takes a foundation of expertise in a topic area, and still then some sweating over. (For some great examples of excellence in infographics, I highly recommend The Visual Display of Quantitative Information, with my favorite Napoleon's March chart.)

Bill Inmon certainly knows a thing or two about data integration, which is perhaps why this picture is so poignant.  And while it is not about EII per se, it reminded me of many conversations I've had with ESB proponents over the past year.  The point I always try to impress on people is that ESBs are good at reliably moving data, and they are (thankfully) more open than some of the older bus technologies, but they still don't help you integrate the data.  The reply to which - and I always love this one - is "All you need to do is write some code..."  If I had a dollar for every time I heard that one...

Here's a better alternative.  Create a parameterized View of what you want in an EII product like Ipedo's.  A View is essentially a virtual representation of the data you want, joined, cleaned, and - importantly - semantically correct.  Then, instead of the typical bus flow of Get A [invoke custom code] then Get B [invoke custom code] then Get C [invoke custom code] and so on, the View queries multiple data sources and returns the information your application needs to the bus.  It's not magic, just a cleaner, easier approach with a more suitable tool.

So before you jump on the bus bandwagon, take another look at that picture and think about how EII can help.

 

Data Integration in a Software as a Service World – Time for Data Services

What does data integration look like when you’re inside a Software as a Service (SaaS) world? Some things look very much the same, and some things look quite a bit different.

What looks the same are the issues. Simply put, how do I get data into the application, how do I get information out of the application, and how do I make sure that it’s the right information? The last item can often be the trickiest, dealing with semantics and data quality issues.

The ways you get data in and out, and how you deal with data quality, look quite a bit different. Most applications offered as a service have a set of APIs available for integration, and most are available as some kind of Web Service. Though Web Services have been around for a few years now, the average IT department does not have the same level of experience with them. So we have a whole new paradigm for dealing with applications that looks a lot different than using an ETL tool, building a data warehouse, or using the TIB.

The other thing to think about is that SaaS applications do not live in a vacuum. The main driver is to connect existing applications (Software as Itself?) with SaaS applications. So now we have to combine old school integration (ETL, EDW, TIB, SQL) with the newer SOA-based integration. Not as easy as it may sound. (In fact, one customer mentioned that the group dynamics among Salesforce.com admins, SOA gurus, and the EDW crew is a challenge in itself.).

All of this got me to thinking that this may be the perfect time for ‘Data Services.’ The term has been gaining momentum since Gartner started talking about it a while back. It’s what we do here at Ipedo.  The basic idea is to take integration functions and make them available as a service via a platform. So the goal is not to move bulk data in batch, but to expose information and make it accessible on demand.  Included in the data service are security, performance optimization, semantic mediation, and potentially even data quality.

To make things more concrete, here are a few examples. You want to combine information from Oracle, DB2 and an order management application available as a Web Services, apply some logic, and update a customer record in Salesforce.com. Right there you have two kinds of SQL, SOAP/WSDL, and the Salesforce.com API (SOQL) to deal with. Or go the opposite direction. Your company is moving from Siebel to Salesforce.com, and you still need to create those weekly reports in Crystal. So you need to deal with the Siebel API or data mart, the Salesforce.com API and somehow get it to output in SQL (and it would be nice if you didn’t have to start from scratch).

There are more examples, to be sure. But one thing becomes clear to me. In the future, there may be no way stations for data. It’s needed inside the firewall, outside the firewall, in the applications you own and the ones you rent. So it would seem data services offer the perfect solution.

Software as a Service requires ‘Integration as a Service,’ aka Data Services. But unlike SaaS, Data Services also need to deal with the past twenty years of application data inside your company. So while I’ll never bet against there someday being a new startup called Integration.com, for the foreseeable future a Data Services platform inside your firewall that can deal easily with both internal applications and rented SaaS applications seems like just the ticket.

I’ll be digging into this idea in more depth. Watch this space.