The Long Tail of Data

This week I turn over the keyboard to my colleague Peter Spielvogel so I can take a long needed summer vacation.  I asked him to write down something he mentioned to me a few weeks ago that I thought was pretty interesting. - T.M.

by Peter Spielvogel

     An article in the business section of today’s San Jose Mercury News on the concept of “The Long Tail” (free, but registration required) made me think about the latent value in the long tail of corporate data. In summary, the “long tail," a term popularized in an article in Wired Magazine, refers to the right section of a steeply declining curve that approaches the X-axis. The premise is that there is money to be made in selling items that are not necessarily best sellers, assuming the carrying costs are not too high. In one example the article cited, Apple’s iTunes site carries 50 times more titles than Wal-Mart. In another, sales of books on Amazon.com that are not even sold in a typical book store comprise the majority of revenue.

     How does this pertain to enterprise information? Certainly, much data resides in relational databases that people and applications access frequently. This "best selling data" is the analog to the top selling books or CDs mentioned in the article, and is likely "popular" because the data is relatively straightforward and easy to query using SQL.

     But, what about the data in the long tail? If ways were available to reach into the other corporate data beyond the reach of SQL, would that increase its use? Unstructured or semi-structured data sitting on isolated servers or individual PCs could contain valuable information about trends or history that might hold the answers to interesting questions. New technologies such as XQuery provide the ability to reach into these areas and allow people and applications to use this data to make decisions.

     Where will Web Services fall in the data graph - on the left among the best sellers or off in the long tail? Or, will Web Services sort themselves out into the popular (stock quotes, interest rates, weather, zip code finder) and the obscure (e.g. lotto number pickers, translate “hello” into different languages).

 

     Data integration solutions such as EII give companies a low-overhead way to extract value from all their enterprise data. They can query any source to get the information they need, without moving or replicating the data. With access to all their data, corporate decision makers have access to information with the breadth of iTunes at Wal-Mart prices.