Balancing Performance in an EII Server

After many years of explaining all of Ipedo's performance optimization functionality bullet by bullet, I finally put it into a picture.   The idea is that you always need to balance the user response time with the impact on the back end data sources.

So, that's all I'll say.  See if this picture does the trick in explaining it.

 

Unstructured Business Content in BI

An interesting article from Colin White over at the Business Intelligence Network called Using Unstructured Business Content in Business Intelligence.  Colin does his usual thorough job, and I have to say the term unstructured business content is pretty nice.  It really helps me separate the useful stuff from what most of us have just been calling "unstructured content."  I think unstructured business content does a better job capturing what's inside industry-specific XML schemas like XBRL, ACORD and others.

And here's a great summary of how this type of content might be used:

A review of semi-structured information shows that a high percentage of this type of information is in an XML format. Tags in XML files provide some semantic information about the contents of the file. There are also an increasing number of industry XML vocabularies, or metamodels, that add additional semantics to XML documents. An example here is XBRL for reporting financial information. XML is becoming the standard approach for exchanging information between systems and between companies.

Examples of applications that can analyze unstructured and semi-structured content and thus enhance BI processing include customer and market intelligence, pricing optimization, customer sentiment and complaint analysis, product safety and quality analysis, regulatory compliance, legal discovery, fraud detection, financial analysis, and IP protection. 

One thing I would like to throw into the mix here is the power of XQuery, especially when it can be easily combined with SQL.  Colin has an extensive section in the article covering the different ways of processing business content, but it comes up a bit short in getting specific on what you can do with XQuery.

First, as opposed to simple search (which he mentions), XQuery can pinpoint, extract and transform relevant pieces of unstructured business content.  Second - and this is the big one - you can do selects and joins with XQuery.  Last, but not least, as we do in Ipedo's product, you can seamlessly combine SQL with XQuery - so creating Oracle + XBRL reports are a snap.  Imagine GROUP BY functions across XML policy reports and your CRM system, all in SQL (with XQuery invoked for you), and plugable right into Crystal Reports.

It's all here today.  In production.

 

Teasing Out 'Integration 2.0'

Those of you who have read my stuff know that this '2.0' bug has been knocking around my brain recently.  It's not so much a cheap desire to gather more buzz for Ipedo (although, by way of disclaimer, I should point out that this is what I am paid to do for my day job), but rather something that keeps nagging me to ask if there's not something to it.

Tim O'Reilly is most famous for declaring Web 2.0 in his article What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software.   His article tried to define what was a very elusive and potentially out of control concept, on the way to becoming a meaningless buzzword.  Along the way he discovered a set of design patterns that were undeniable.  Not that everything was totally new - there were people doing 2.0 stuff in the 1.0 era - but overall there was a pattern.  I liked the following way of describing Web 2.0:

You can visualize Web 2.0 as a set of principles and practices that tie together a veritable solar system of sites that demonstrate some or all of those principles, at a varying distance from that core.

We are on the verge of 2.0 everything.  I see many of these as hollow marketing efforts.  On the other hand, I see 2.0 as a challenge - a new way of thinking.   Our new 'ng' or '.next.' 

So far, the integration vendors have not really done anything to define what the next generation of integration should look like.  Saying that it's SOA is a punt.  That's not an answer.  The question is whether there is something new, something that can be hosted, a service, collaborative, open that will cost less and yield more.

Stay tuned here as we ponder.

Thoughts on 'BI 2.0'

Came across an interesting post Tom Hudock's Business Intelligence for Business People blog.  The post is called 'Explain Why BI 2.0,' and as the title suggests, prods at the idea to tease out the real meaning.

Not being satisfied with the definition in IDC's report, which states "BI 2.0 is for customers interested in providing their employees with advanced decision support that solves specific business problems," Tom goes on:

That's not really descriptive enough,  so my BI 2.0 would:

• track business events
• make decisions in close to real-time
• use SOA and Web 2.0 technologies

I agree with Tom that IDC's definition is way too high level.  And I generally agree with his additions to the definition.  But in my opinion, something crucial is missing.

Whether you like the "2.0" christening trend or not, one thing that is common to all is an increase in user-created content and relationships.  I would say whether your 2.0 app is built using XML or JSON, AJAX or Ruby, HTTP or RPC, what matters is that users are more empowered to create.

So I don't know exactly what BI 2.0 is, but seems to me that more power needs to be put in the hands of the users.  When it comes to data, that means more power to get data, slice and dice it as needed, then publish and share the analysis.

This is not simple matter, and I can sense the raised hackles of data architects already, but the reality is that unless we can simplify the gathering of relevant data and put it into the hands of the users, we will never have BI 2.0.  IMO, we need an Integration 2.0 before we can have a BI 2.0.

Stay tuned for thoughts on that.

EII, SOA and Virtualization: Roundtable Podcast from Dana Gardner

I've been waiting to use the category tags EII, SOA and Virtualization together for some time now.  It's something readers here have heard me talk about before - the nexus of the SOA trend and the virtualization trend, and how EII nicely joins the two together when it comes to data.

The podcast is a lively roundtable format with Steve Garone, Joe McKendrick, Neil Macehiter, Tony Baer, Jim Kobielus and Dana Gardner - some pretty knowledgeable folks.  Though the podcast starts out with a debate on SOA stack vs. best of breed implementations, it gets more interesting toward the end when they delve into other subjects.  I must say, it's nice to see EII come up in the context of a general SOA discussion.  They definitely get the importance of data virtualization in an SOA in general, and how EII can handle data and master data nicely.

Listen to it here.

Database Virtualization for Business Continuity

I happened across Ken Oestreich's Fountainhead blog, where he recounts the following from a conversation with a sales rep:

"He's noted that "Virtualization" is being inaccurately  subsumed to mean the software hypervisors purveyed by VMware.

You know, I couldn't agree more.  VMware is certainly the best known virtualization product, and a good jumping off point for discussions, but I think people need to understand that this is a bigger trend, as IDC is covering.

The problem with only understanding server virtualizaiton is that you limit your thinking to hardware consolidation.  And there are so many other benefits.  One that I've been paying attention to recently is business continuity (nee disaster recovery).

When you think about it, this is a great fit for database virtualization.  What happens when my database goes kaput?  When you virtualize, you have options.  You can redirect to another database.  Or, you could even have a backup Web Service that would provide the data (perhaps master data, position data, sales data).  The virtualization layer insulates the users and applications from outage.

Last year, Ipedo introduced a feature called dynamic failover redirection, where Ipedo XIP can automatically look to a pre-defined replica if the primary data source is not available. This boosts uptime and minimizes business disruption by delivering real-time information to users from wherever it resides. 

Think of it as failover at the the view level - read virtual level - rather than at the database level.  Another interesting aspect of the ever expanding world of virtualization.

What's Next for Virtualization? Databases. Just Follow the Money.

I've been writing a bit recently about virtualization, specifically how database virtualization works.  And in my quest to get up to speed on this new topic, I've been poking around a number of blogs that cover virtualization.  Today I came across InfoWorld's Virtualization Report, where David Marshall ran down IDC's predictions for what's next in virtualization.  You can listen here.

I find the predictions perfectly reasonable, but it was what I didn't hear that got me to thinking.  No one is talking about virtualizing databases.  Granted, I have an agenda (find me a blogger who doesn't).  I'm already convinced database virtualization is the next big thing.  So let me try and lay out a way of thinking about this from a customer perspective.

The main driver of virtualization is cost reduction, right?  So vendors and IT shops have attacked the very expensive areas of servers and storage.  Makes sense.  So what's another commoditized area that companies spend tons of money on, and that keeps getting bigger and harder to maintain every year?  Answer: Databases.

So I wonder why more attention is not being focused on this area.   Perhaps we vendors have not done a good job explaining it.  Database virtualization reduces the cost of maintaining dozens of custom data marts, allows movement of older databases to commodity hardware (or retirement altogether), and reduces the costly and time-consuming process of copying data.

In addition, database virtualization provides more flexibility in deployment (another oft-touted benefit of virtualization) and rapid response to change (yet another).

For the database guys who claim that databases are a horse of a different color, too complex to be messed with, I ask you: What's the hottest trend these days in the database world?  Master data management, right?  Providing a virtual layer to control schemas for things like customer, partner, account, etc. seems just the ticket.  And for those who say it will never be fast enough, we know that's just not true.  First, not every database needs to be a screamer.  And for the hot data (see my previous post on the temperature of data), just cache it. 

So think about it.  I'll bet you $100 that 100 out of 100 CIOs would love to reduce the number of databases in their organizations.

What's next for virtualization?  I say databases.  Just follow the money.

A Discussion on Virtualization with Stephen Hayward

Had a good conversation with Stephen Hayward of ProjectX around data virtualization.  Stephen did a good job summarizing the discussion and providing some background, which you can see in this entry  Or go right to the podcast.

What Does Data Temperature Mean, and What's It Got to do with Data Virtualization?

I was reading a post from Dan Linstedt from last year on the temperature of data.  This is a very interesting concept, and so intuitive I wondered why I had never heard it before.

You should read the whole of Temperature of Data for RDBMS, and DW 2.0, but here are Dan's definitions, summarized:

• Hot = accessed all the time, or extremely important data requiring sub-second response times
• Medium = is data accessed most of the time, but where response times can be anywhere from 1 second to 10 or 15 seconds
• Lukewarm Data = data that is accessed rarely; rarely may be (for example) once every 30 minutes or twice every 4 hours. 
• Cold Data = historical context that is hardly ever accessed, but when requested, must have a response time equal to that of a couple minutes

What's really interesting about these ratings is that they can serve as a guide to proper data placement, and hence can inform how one should use data virtualization. 

If you have data that's Hot, it should either be cached, or the application should hit the source directly.  If you have data that is Medium, then it need not be cached, since query federation can easily hit these time frames.  Likewise with Lukewarm data.  Cold Data might even be put off to tape or cheap storage with a low query priority and a never-cache setting.

As Dan points out, definitions will vary by company (and maybe even within companies), but the framework can be universally applied.  We'll continue to dig into this concept and show how it could work in practice.  Think about it, and I'm sure you'll warm up to the idea.

New XQuery Views

For those that missed it, Ipedo announced some exciting new XQuery View capabilities last week, just as XQuery 1.0 became an official standard (you can see my entry on that here.)

Anyway, I thought I'd summarize what we announced.  We think XQuery is going to have an even bigger impact now that the "When will it become a standard" question is out of the way.  Here are the high points:

• Processing of XML Stored in Relational Databases - Ipedo's XQuery can be used in conjunction with the XML processing and XQuery now available in Oracle 10g, DB2, and Microsoft SQL Server.
  
• XQuery View Caching - Ipedo XIP allows XQuery Views results to be cached natively in XML, providing both faster access for users, and post-processing and drilldown using XQuery.
  
• Streaming Result Processing - The new XQuery engine implements query operators that stream data values, ensuring that query evaluation is memory efficient and allowing clients to cursor through results at the lowest computation costs.
  
• Publishing of XQuery Views as Web Services - XQuery Views can now be invoked directly via a Web Services API, making it easy to create advanced data services. Ipedo XIP can automatically generate the WSDL file needed for a particular XQuery View.
  
• Extension Mechanisms for Custom Data Sources - In addition to standard data sources, Ipedo's XQuery extension framework now makes it possible for users to make custom data sources, such as in-house applications accessed using custom APIs.

Srinivas Pandrangi shows a simple example of using XQuery Views in his latest post over in The EII Files.