BI and Enterprise 2.0

Came across an interesting article from Colin White, who has been chronicling changes in the BI and integration worlds for some time now.  It's called Using Enterprise 2.0 for Business Intelligence, and worth the read. 

Here are some of the points that stood out for me:

Enterprise 2.0 does not replace current approaches – it simply provides innovative ways of quickly building some urgently needed business user capabilities.

This is a great opening point.  Business users and BI analysts should agree that bringing in some more modern Web features to BI is a good thing.  CIOs might like the fact that they can do it on the cheap.  Existing BI vendors might not like this (more on that later).

Most business intelligence applications focus on creating reports and analytics that aid executives and analysts in developing and optimizing strategic and tactical business plans and initiatives. As I have discussed in previous Business Intelligence Network articles, companies now want to take business intelligence to the next level by using it to drive daily business operations and to expand its use to a broader set of users, both inside and outside of the organization.

Another good point, something many of us have been calling "BI for the masses."  Some have derisively termed this "BI for the unwashed masses," feeling that not everyone can understand what they are seeing.  This is certainly an unenlightened view, and a dangerous one if your competition puts information into their customers' and employees' hands.

And what does the technology look like.  Here are three of my favorites, that fall in line with what we've been talking about here at Ipedo for some time:

1. Information Exploration and Analysis: Employs technologies such as federated queries, enterprise search, and content analytics to explore and analyze unstructured and semi-structured business content, including that produced by the information collaboration component.
    
2. Information Integration: Extends traditional enterprise data capture and transformation technologies with the ability to process unstructured and semi-structured business content, including that produced by the information collaboration component.   
    
3. Information Syndication and Delivery: Uses syndication protocols (RSS and ATOM, for example), syndication servers, and data mashups to publish and deliver all types of structured information and unstructured business content to enterprise applications and users.

And he closes with this synopsis of what the current BI vendors are doing:

Enterprise 2.0 approaches have started to gain momentum outside of the business intelligence environment, but BI vendors have been slow to consider, or adopt, new Web 2.0 technologies. This has opened up an opportunity for smaller vendors outside of the traditional BI marketplace to compete with established BI vendors for a slice of the Enterprise 2.0 BI environment. These new vendors will be able to move quickly into the BI marketplace with lower cost and more usable solutions that may not have the functionality of the big BI platforms, but nevertheless will be attractive for addressing certain types of business problems. Meanwhile, the large BI platform vendors will find it difficult to move rapidly to compete because of their poorly integrated and older product architectures. It will be interesting to see how this battle will emerge over the next two to three years.

This is certainly true.  Why, one large BI vendor who shall remain nameless is still built on CORBA.  CORBA!  How many times have you seen that on the agenda at a Web 2.0 conference?  The point is that many product suites are showing their age.  In most cases, a more modern approach can yield just the results you need.

Fusing Together Oracle Products with Data and Service Virtualization

We did a demo recently that I thought was worth sharing.  With all the talk of Oracle Fusion in the press recently, I thought it was worth showing how simply and easily you can create custom combinations of data using Ipedo's data and service virtualization techniques.

In the demo we fuse together data from some Oracle 10g databases, with Salesforce.com (ex-Oracle CEO,  so we threw it in) and data moved from SQL Server by Oracle DI.  We put it all into Oracle BI Publisher, run Ipedo on Oracle AS, use another Oracle 10g database as a cache, and lastly move it into Oracle BPEL.  Whew!  Our professional services guys are now almost Oracle experts.

Here's the demo.  It's 15 minutes long, but broken into sections so you can navigate around quickly.

Fusing SQL and XQuery to Combine XML and Relational Data for Reporting

There is a lot of talk these days about harnessing content for use in business intelligence.  This is something we've been enabling for some time at Ipedo, but it seems like more and more people are having the need/becoming aware of the possibilities.

Generally speaking, there are two types of content people are interested in - search results from textual documents and access to documents represented as XML.  I know, I know, there are more than that.  But if you look at what people are actually using, I contend they are looking to utilize search results on Web and/or Microsoft Office documents; or they want to search and extract information from semi-structured XML documents, which are usually forms, industry schemas (MISMO, FpML, ACORD et al), or transformed Microsoft Word or Excel.

So I want to show you something really cool.  Take a look at the picture below, which shows how you can use SQL to query semi-structured XML documents.  Perfect for BI.

 

The magic is our Dual-Core Query Engine that marries SQL and XQuery.  We have something called XML Tables that make XML documents look like relational tables.  So you can query a bunch of XML documents using SQL and manipulate it in a reporting tool like Crystal Reports.  Or you can query a combination of relational and XML sources using SQL and manipulate in Crystal et al.

If you want to see something live, you can also see how this same technology works against combinations of relational and Web Services data here.

Unstructured Business Content in BI

An interesting article from Colin White over at the Business Intelligence Network called Using Unstructured Business Content in Business Intelligence.  Colin does his usual thorough job, and I have to say the term unstructured business content is pretty nice.  It really helps me separate the useful stuff from what most of us have just been calling "unstructured content."  I think unstructured business content does a better job capturing what's inside industry-specific XML schemas like XBRL, ACORD and others.

And here's a great summary of how this type of content might be used:

A review of semi-structured information shows that a high percentage of this type of information is in an XML format. Tags in XML files provide some semantic information about the contents of the file. There are also an increasing number of industry XML vocabularies, or metamodels, that add additional semantics to XML documents. An example here is XBRL for reporting financial information. XML is becoming the standard approach for exchanging information between systems and between companies.

Examples of applications that can analyze unstructured and semi-structured content and thus enhance BI processing include customer and market intelligence, pricing optimization, customer sentiment and complaint analysis, product safety and quality analysis, regulatory compliance, legal discovery, fraud detection, financial analysis, and IP protection. 

One thing I would like to throw into the mix here is the power of XQuery, especially when it can be easily combined with SQL.  Colin has an extensive section in the article covering the different ways of processing business content, but it comes up a bit short in getting specific on what you can do with XQuery.

First, as opposed to simple search (which he mentions), XQuery can pinpoint, extract and transform relevant pieces of unstructured business content.  Second - and this is the big one - you can do selects and joins with XQuery.  Last, but not least, as we do in Ipedo's product, you can seamlessly combine SQL with XQuery - so creating Oracle + XBRL reports are a snap.  Imagine GROUP BY functions across XML policy reports and your CRM system, all in SQL (with XQuery invoked for you), and plugable right into Crystal Reports.

It's all here today.  In production.

 

Thoughts on 'BI 2.0'

Came across an interesting post Tom Hudock's Business Intelligence for Business People blog.  The post is called 'Explain Why BI 2.0,' and as the title suggests, prods at the idea to tease out the real meaning.

Not being satisfied with the definition in IDC's report, which states "BI 2.0 is for customers interested in providing their employees with advanced decision support that solves specific business problems," Tom goes on:

That's not really descriptive enough,  so my BI 2.0 would:

• track business events
• make decisions in close to real-time
• use SOA and Web 2.0 technologies

I agree with Tom that IDC's definition is way too high level.  And I generally agree with his additions to the definition.  But in my opinion, something crucial is missing.

Whether you like the "2.0" christening trend or not, one thing that is common to all is an increase in user-created content and relationships.  I would say whether your 2.0 app is built using XML or JSON, AJAX or Ruby, HTTP or RPC, what matters is that users are more empowered to create.

So I don't know exactly what BI 2.0 is, but seems to me that more power needs to be put in the hands of the users.  When it comes to data, that means more power to get data, slice and dice it as needed, then publish and share the analysis.

This is not simple matter, and I can sense the raised hackles of data architects already, but the reality is that unless we can simplify the gathering of relevant data and put it into the hands of the users, we will never have BI 2.0.  IMO, we need an Integration 2.0 before we can have a BI 2.0.

Stay tuned for thoughts on that.

What Does Data Temperature Mean, and What's It Got to do with Data Virtualization?

I was reading a post from Dan Linstedt from last year on the temperature of data.  This is a very interesting concept, and so intuitive I wondered why I had never heard it before.

You should read the whole of Temperature of Data for RDBMS, and DW 2.0, but here are Dan's definitions, summarized:

• Hot = accessed all the time, or extremely important data requiring sub-second response times
• Medium = is data accessed most of the time, but where response times can be anywhere from 1 second to 10 or 15 seconds
• Lukewarm Data = data that is accessed rarely; rarely may be (for example) once every 30 minutes or twice every 4 hours. 
• Cold Data = historical context that is hardly ever accessed, but when requested, must have a response time equal to that of a couple minutes

What's really interesting about these ratings is that they can serve as a guide to proper data placement, and hence can inform how one should use data virtualization. 

If you have data that's Hot, it should either be cached, or the application should hit the source directly.  If you have data that is Medium, then it need not be cached, since query federation can easily hit these time frames.  Likewise with Lukewarm data.  Cold Data might even be put off to tape or cheap storage with a low query priority and a never-cache setting.

As Dan points out, definitions will vary by company (and maybe even within companies), but the framework can be universally applied.  We'll continue to dig into this concept and show how it could work in practice.  Think about it, and I'm sure you'll warm up to the idea.

Calculating the ROI of EII: Application Maintenance (Example 5/5)

Because the EII layer can easily handle additions or changes to underlying data sources, organizations can focus on maintaining the EII layer and not the maintenance of every application and its connection. If a data source changes, the administrator only needs to update the mapping in the EII layer; all reports remain unchanged as they use the schema defined in the EII layer.

To figure out if this can save you money, think about this question: How much does it cost to maintain point-to-point connections among all your applications and data sources?

                                                                                         

	Example	Formula
Conventional   Approach
Cost to   develop a customized connector between two applications ($)	$10,000	A
Number of   connections required per year	25	B
Application   maintenance cost using conventional approach ($)	$250,000	A * B
Cost to change   a report because a data source changes ($)	$1,000	C
Number of   changes to reports required per year	200	D
Application   maintenance cost using conventional approach ($)	$200,000	C * D
EII   Approach
Cost to   develop a customized connector between two applications ($)	$0	E
Cost to change   a report because a data source changes ($)	$0	F
Cost to change   a data source mapping in the EII layer ($)	$100	G
Number of   changes to reports required per year	200	D
Application   maintenance cost using EII approach ($)	$20,000	G * D
Potential annual   cost savings from EII ($)	$430,000	(A * B) + (C *   D) - (G * D)

In this example, a company that eliminates customized programming for connecting applications and changing reports will save $430,000 by using EII.

• What value-add projects could your IT team complete if they are not changing report formats?
• How much more motivated would your developers be if they spent less time on software maintenance?

That concludes our five-part analysis.  Hopefully you've found some areas where there might be cost savings.  If you did and would like to take things further, here are some suggestions:

1. If you want to use your own numbers in the above examples, you can request the companion Excel spreadsheet from Ipedo.
2. If your business situation requires a customized financial model, one of our analysts can provide assistance with the process, or even build an interactive spreadsheet based on information that you provide.
3.  Once you complete your financial models and convince your team that EII looks good on paper, a logical next step is to develop a proof-of-concept application that demonstrates cost savings and/or productivity improvement in a controlled environment. Ipedo can help with this as part of the sales process.

Calculating the ROI of EII: Data Consolidation (Example 4/5)

Managing data costs a lot of money.  Just ask any CIO.  In any given company, there is a lot of money spent (and wasted) moving and copying data.  Enter EII.

Using EII to create a virtual, centralized database that cuts across disparate systems can reduce the cost of data maintenance, and eliminate the need for additional investment to store replicated data. Even with a conservative estimate of $50K per data mart ($25K hardware purchase and software licensing, $10K development cost, $5 software maintenance license, $10K personnel to manage the data mart), this gets expensive quickly.

The ROI-unlocking question: How much of your IT budget goes towards moving data around the enterprise and storing copies for different applications to use?

                             

Top-Down   Approach	Example	Formula
Annual IT budget   ($)	$10,000,000	A
% of resources   spent on data migration and replication (%)	25%	B
% improvement   with EII (%) (estimate)	10%	C
Potential   annual cost savings from EII ($)	$250,000	(A * B) * C

In this example, a company that spends 25% of its $10 million IT budget on data migration and data replication would save $250,000 (2.5% of overall IT budget) by using EII to complement its other data integration initiatives.

 

                       

Bottom-Up   Approach	Example	Formula
Cost to deploy   single new data mart ($)	$50,000	D
Number of data   marts deployed per year	10	E
Potential   annual cost savings from EII ($)	$500,000	D * E

In this example, a company that can eliminate 10 data marts per year by using EII will save $500,000.

And with all the focus on CDI and MDM these days, these upside benefits are sure to resonate:

• What is the value of having a “single version of the truth” based in the original data sources, rather than copies of data that become obsolete once the source changes
• How much time do you spend reconciling different versions of data residing in different locations?
• What politics are involved in getting permission to replicate data in your local data mart?
• How could you better deploy your IT resources if they did not have to manage a data mart?

That's it for today.  Next time, our final installment.

EII Haiku

In high technology, we often do things that have never been tried before.  I'm fairly certain this is one of them. 

Since I was recently accused of being too serious with my ROI posts, but not wanting to give way to the senseless prattling of many blogs, I offer you EII Haiku.  I like to think of them as lighthearted but deep.  Enjoy.

Queries federated
Distant rows and columns meet
My warehouse unscathed

Web Services, SOAP
Loosely coupled, lightly known
Join them, EII

Data is scattered
Metadata, SQL meet
All becomes as one

P.S. - If you would like to share your own, please feel free to do so in the Comments section.

Calculating the ROI of EII: Increased Flexibility (Example 1/5)

As we move into the planning and budgeting season, the time seems right to start talking about ROI.  At Ipedo, we've spent quite a bit of time analyzing how our customers can save money and improve productivity using EII.  In this five-part series, I'll show you how to do it.

What kicked off our study of the financial benefits of EII was a 2005 study by The Data Warehousing Institute (TDWI).  In it, 69% of respondents said that data integration issues are a barrier to implementing new applications. The three main issues they listed were poor data integration infrastructure, lack of a business case and inadequate funding, and data quality and security.

This series will help decision makers responsible for data integration projects develop a business case that quantifies the potential cost savings of an EII project based on their organization’s specific information integration needs.  Each installment will focus on a specific area where EII can generate cost savings and productivity improvements:

• Increased flexibility to respond to changes in IT requirements
• Better analyst utilization
• Higher developer productivity
• Lower data consolidation costs
• Less expensive application maintenance

A few housekeeping items before we start.  In the  scenarios, we provide examples to illustrate the potential cost savings. Each example is for a departmental EII implementation, as opposed to an enterprise-wide deployment. Most organizations start with a single use for EII and then scale it to different departments once they achieve their target results and become comfortable with the technology.

The cost savings numbers represent the ongoing annual savings, after the initial cost of implementing the EII system. This cost of an EII system will vary, depending on your specific needs. For the departmental-scale examples in this document, a 1- or 2-CPU system with matching failover server will be sufficient in most cases. A complete Ipedo deployment for a scenario, including software license, maintenance, training, and some customization through our professional services group will cost around $200,000. A bare-bones EII system can cost as little as $50,000.

Your savings may be higher or lower than what we illustrate, depending on your specific use case and the cost of implementing your EII solution. To simplify testing different scenarios, running your own numbers, and performing “what if” analysis to determine your own cost savings, Ipedo has a companion Excel worksheet. If you are interested, let me know and I can send you a copy.

So, with all that out of the way, let's get down to brass tacks and look at how Ipedo can save you money by increasing flexibility in a reporting application.

Increased Flexibility in Reporting

In today’s information economy, having access to data across the enterprise gives managers and executives a significant advantage in making real-time decisions that affect their business. EII allows companies to respond quickly to changes in market conditions without incurring a major IT investment.

What effort is involved to link new data sources to your existing reports or applications?

	Example	Formula
Conventional Approach
Cost to modify a report to integrate a new data source ($)	$1,000	A
Number of changes required annually	200	B
Cost to modify an application to integrate a new data source ($)	$5,000	C
Number of changes required annually	50	D
Reporting cost using conventional approach ($)	$450,000	(A * B) + (C * D)
EII Approach
Cost to modify a report to integrate a new data source ($)	$100	E
Number of changes required annually	200	B
Cost to modify an application to integrate a new data source ($)	$0	F
Number of changes required annually	50	D
Reporting cost using EII approach($)	$20,000	(E * B) + (F * D)
Potential annual cost savings from EII ($)	$430,000	(A * B) + (C * D) - (E * B) + (F * D)

In this example, a business unit that modifies four reports and one application a week will save $430,000 by using EII.

Additional Upside

These are a few soft benefits.  You can either try to assign a value, or you can just treat these as icing on the cake.

• By getting useful information sooner, what is the value to your organization?
• What new insights can your managers discover by having access to a unified view of disparate information that EII provides?
• As new regulations appear, how quickly can you respond to generate the necessary documentation? What is the burden on your organization to create these reports? Whose budget covers producing regulatory reports?

That's all for now.  I'd be curious to hear back on how people think these numbers match their own situations.

Stay tuned for our next installment.