EII Exposed: The 10 Hidden Complexities

Director of Product Management Peter Spielvogel weighs in on a recent article. -- T.M.

I recently read Tieu Luu’s article on “Ten Things Your Enterprise Information Integration Vendor May Not Tell You: The hidden complexities of implementing an EII solution.” He has captured many of the key implementation barriers that can trip up organizations with unrealistic expectations about EII, often fed by vendors who are either naïve (best case) or deceptive (worst case). Real-world EII deployments are often quite different from the sanitized success stories that appear in most articles on the subject.

In his summary, he wisely suggests that companies considering EII take the time to thoroughly understand their current data environment, and application’s needs. I would add that they should work with their EII vendor(s) to create a proof of concept application that connects to a representative set of data sources, running queries comparable in complexity to what they would expect in a production environment.

While I’m sure some of the “surprises” he discusses do occur, in this blog I will address how Ipedo handles these challenging real-world EII implementation issues.

1. "It's probably a lot harder to implement an EII solution in your enterprise than our examples and case studies would tell you."

Most companies we talk to realize this.

For example, one organization’s VP of Development placed a prominent note on their proposed architecture diagram: “When source data is dirty and/or inconsistent, data access layer construction will encounter the same class of problems as the ETL steps of a data warehouse. There is no magic.” He clearly understands where EII will fit in his overall data integration solution.

Another customer originally expected an out-of the box solution to the foreign key across different database issue. After a brief discussion, they agreed that mapping these relationships manually made sense, especially since they only had to perform that process once.

2. "We isolate you from all of the complexities of accessing multiple data sources... including the ability to debug."

Ipedo’s ‘Explain Plan’ shows every step of the integration process, including the SQL that is passed to each data source. This allows administrators to trace individual queries and streamline the debugging process.

3. "Be prepared to lose some platform-specific functionality and flexibility."

Ipedo addresses this by allowing users to pass vendor-specific SQL to the back-end database. Allowing proprietary extensions to standard SQL balances the advantages of a common interface with the power to leverage platform-specific features.

4. "Sure we support transactions, but only if they go through a single data source."

5. "Of course we support write operations, but you may not actually be able to use them."

Luu’s advice here is perfect: “understand what type of distributed transaction support your enterprise will need.” In some cases, read-only access will be fine; in others the ability to update a single data source will suffice. Some organizations will need full XA support. Ipedo’s vast transaction processing experience allows us to help our customers design the right system architecture that exploits the capabilities of the EII layer and complements it with robust application code to address idiosyncrasies of individual back-end systems.

6. "Our product can be clustered, but that doesn't guarantee you'll have proper backups."

Supporting distributed high-availability environments is one of the challenges of deploying EII in production environments. As with any database cluster, when the primary server goes down, the application will need to switch over to the failover server. Ipedo has used many techniques, both in the product and in the operating environment, to address this problem. The key is to start with the specific customer needs and use both the EII layer and operating environment to create a workable solution. For reporting applications that use EII for read-only access to data sources, this is less of an issue.

7. "Tuning this thing for performance can be a nightmare."

Performance depends on the back-end data sources, types of queries you want to run, and the optimization algorithms the EII layer uses. Ipedo looks at the source data to determine its join algorithms (cost-based optimization) rather than relying only on pre-set formulas (rule-based optimization). Ipedo also uses caching to speed performance. Of course, caching is subject to the caveats Luu lists in item #10.

8. "Better make sure you set aside some time and money to build the proper security infrastructure if you really want to use this thing."

An EII solution must fit into the customer’s existing security model and not attempt to reorganize the entire data security model around its own needs. For example, some organizations want the EII layer to store and use access to remote data sources while others expressly prohibit any persistent storage of login information. Ipedo allows granular security of data sources, tables, and views by individual users and groups at the EII layer as well as connections to single sign-on systems, proprietary authentication systems, and whatever other safeguards were in place prior to the EII deployment.

9. "Say goodbye to the concurrent access behavior that you've been used to."

The issues Luu raises are associated with all transactional databases. So, for transactional EII, they remain a significant issue. For analysis, reporting, portal, and dashboard applications, which are less transactional, most companies will not have to worry about this issue. If you are planning on using your EII layer to support OLTP (few of our customers do), you should be careful in your application to use the correct isolation level.

10. "Unlike ETL products, we offer a federated approach that doesn't store any data, sort of..."

While EII generally promises data on-demand, caching is commonly used to boost performance. In the EII context, caching refers to storing only query results, and not moving entire data sets. To avoid the synchronization problems that Luu describes, Ipedo uses intelligent caching that automatically invalidates itself at pre-defined intervals. In addition, Ipedo allows applications to force a query to go against the original data source if the application requires “fresh” data.

Summary

Tieu Luu’s article demonstrates the growing sophistication in the EII market and the consultants, analysts, and journalists that follow it. Customers should ask difficult questions and demand to see proof points in their own environments. Only then will EII become another standard information integration tool that organizations rely on to make their operation business decisions.

Additional EII Resources

Ipedo offers several resources to help individuals become more informed about EII and how to use it to drive business results.

EII Guide - provides a pragmatic approach to implementing enterprise information integration in your enterprise. It includes sections on: EII technology, applications where EII can add value, suggestions for starting an EII project.

Determining Financial Returns on EII Projects - How to quantify the potential cost savings of an EII project based on your specific information integration needs.

The annual Ipedo EII Survey polls a broad cross section of IT professionals to determine how they are using EII and their implementation plans for the coming year. To weigh in with your opinion (through the end of the year), please visit the survey start page. After that, the same web link will provide you an opportunity to download the survey results .