What good is an Information stream?

Information flows in Enterprise software architectures present applications with an opportunity and a problem. The opportunity, that is all too often ignored, is that observing these streams in transit can be a source of operational business intelligence. Aggregating, analyzing, interpreting and presenting intelligence from information flows can provide deep insights into the operations of business processes. The problem is one of efficiency and effectiveness; introspecting flows without introducing undue latency and at the same time being capable of performing sophisticated processing on those flows.

 
Ipedo’s XIP addresses a subset of the problem; that concerned with processing XML document flows. Since XML is a popular format for serializing information, a number of applications are found producing and consuming XML flows. XIP provides a construct named Feeds for processing streams of XML documents efficiently. Feeds are composed of one or more Stages, each of which performs a specific type of operation on the stream of XML documents. A number of stages can be chained together to achieve desired processing of an XML document flow.

Here is a screenshot of a feed definition. This feed operates on a stream of ACORD (Insurance industry standard) XML documents. It is composed of three stages - an XQuery stage which checks for the presence of a specific element, and if absent adds in the element with pre-defined default content; an XSLT transformation stage; and an aggregate index (more on aggregate indexes later) stage that keeps track of aggregates that apply to the entire flow.

To create a feed, you start by defining the stages that make up the feed. Each stage is identified by a name, and can be one of several kinds, as seen in the next screenshot.

• XML Rules can be used for semantic/business rule validation
• XSLT transformations can make structural changes to the document
• XQuery can be used to perform both transformations and queries that span the input document and a variety of data sources available to the XIP platform
• The Java DOM Filter can be used for custom processing coded in Java
• Compound Aggregate Indexes (CAI) can be used to keep running aggregates (counts, sum, average etc.) of values found in the XML document stream.

What follows are illustrations of how stages are defined. Here is an XQuery stage which accesses the input XML document to the stage using the input() function and can then use standard XQuery syntax in composing queries that can not only transform that input, but also enhance it on the basis of information queried from any external data source (web services, relational databases, flat files, custom java applications, you name it). These constructs make it possible to extend the processing power of feeds in virtually limitless ways.

Another example is a CAI (Compound Aggregate Index) stage that maintains indexes on aggregates computed on a flow of documents. For instance, you can keep track of the count of documents of a given transaction type, or you can keep a running total of order values for a given customer. These indexes can feed operational intelligence dash boards that will provide real time business monitoring capabilities.

The Feeds feature is interesting to anyone considering adding value to information flows by monitoring them, applying syntactic or semantic validation to those flow units, enhancing the quality of the data by filling in missing information or transforming that flow to match a target applications needs. The possibilities are endless.