As you can read on this link we recently co-sponsored a very successful conference here in New York City. It was a great conference and all attendees reported very positively on the conference. We were very pleased to be a part of it.

At the conference I gave a one hour presentation about the emerging importance of the Power Data Analyst. You can read the presentation on this link.

A key part of our message is how Meta5 is ideal for the “Big Data” environment. One of the defining factors of “Big Data” is that it means “MORE data”. This blog entry will introduce this idea of Meta5 in a MORE data environment. If you would like to watch me give the full presentation again in our office environment then you are invited to watch the video on this link.

Where are We Today?

This diagram gives a good overview of where we are today.

We all know about data warehouse 2.0. Bill Inmons great book describes it in detail so we do not need to tell you all about that. There are a lot of great reasons why data should go in to the data warehouse.

The greater the need for governance and accuracy? Such as ledgers, account balances, invoices, payments? You need those things to be 100% accurate all the time. So they have to go in to the data warehouse. No question.

As the perceived value of the data and the apps increases? Such as statements, invoices, official letters etc? They need to go in to the data warehouse as well. In short, if the data is important enough to build an operational system, or manage it with an operational system, there is a good case that the data should be housed in the data warehouse. No question.

But there is a cost to putting data in to the data warehouse and DW budgets are not infinite. There is also a LOT of data that is being generated by “big data” apps that is not from operational systems that are available to easily extract the data and put it into the data warehouse. The most obvious of such data are mailing lists and prospect lists created by the world of social networks. There are many more sources of prospects today thanks to social media and third party vendors of mailing lists.

Some of this data is so small that it can be readily handled in Excel and Access. Everyone has Microsoft office available to them to handle small amounts of data like this.

What I want to point out is that there is this “middle bit”. And it is growing.

There is data that is coming more and more available that is too large to handle in Excel/Access and similar databases but there is a real question that this data belongs in the data warehouse. Certainly there is a real question as to whether the budget can be found to put this data in to the data warehouse among all the other important work that needs to be done.

So what are the Power Data Analysts of the world to do? Are you to go to IT and argue your case that this “middle bit” of data must go into the data warehouse before you can really tell it’s value? How might you go with such attempts to create business cases where the value of the data is truly unknown? Experience tells us that is a pretty hard sale to make to IT, right?

A More Agile Approach.

What is needed is a more “Agile” approach so that you can establish the value of data that is new to your company before asking for it to be placed in the data warehouse and providing the necessary business case to do so. A more agile approach could be the difference between success and failure.

That more “Agile” approach is Meta5. With Meta5 you, the Power Data Analyst, can bring almost any data known to man in to your Data Analysis Desktop. You can build prototype ETL to integrate it to the data you already have. You can do this in order to generate a more accurate determination of the value of the data before going to IT and submitting your business case to have the data included in the development cycles for the data warehouse.

I like to use examples to explain what I mean. Let us say you work in marketing as a Power Data Analyst. Lets us say a number of vendors, say 5, were offering you customised prospect lists that they created using social networks and opt ins. How could you tell which vendor had the best lists for your company? After all? All the vendors are going to make the claim they are the best vendor for your company. You can’t tell until you try them out.

If you go to IT and ask them to put 5 sources of data into the data warehouse for testing where you know that you will not buy from 4 of them, or maybe not even any of them, you will be told “no” and rightly so. IT time is expensive and is constrained in most organisations. You have to “get in line” to keep IT people at 100% utilisation. You will be told to look for a “cheaper” way to determine which vendor you are going to buy the lists from if you buy from any of them at all. IT might be able to spare someone to give you a bit of help, but they are busy for good reason.

With Meta5 you could ask all 5 vendors to give you samples of their lists according to your specifications. You could then import those sample lists with all the associated data the vendors are willing to give you. You can then integrate it with your known customers and prospects in the data warehouse so that you can do things like exclude any email addresses where your company has been told “do not email”. It is very important that you have an environment that can integrate these external “big data” sources with your data warehouse. Emailing people on a sample list who have already told your company not to email them may end up in lost customers or perhaps even a legal complaint! You do not want to be the cause of such hassles for your colleagues.

Once you have integrated the external prospect sample lists and excluded those prospects that your company rules or guidelines say should be excluded you can run the sample email campaign. As the responses come back through your operational systems, though your data warehouse, you can determine the relative value of the lists from the various data vendors. Perhaps one of them makes the qualification criteria you want for email solicitations. Perhaps not.

What you do know is the actual performance of each of the vendors sample lists using your email marketing message for your products. And you were able to do that without needing IT support to perform the emailing.

By this time, if one or two of the list vendors has proven their worth? You can go to IT with your business case and ask them to design and build a production quality ETL system between your company and the vendor to accept, clean, dedupe, and integrate their prospect list processing with your companies data warehouse. This is usually a significant amount of work that will cost a significant amount of money but that is now well justified by sample emailings and actual sales results.

You can then work with the list vendors over time to optimise the algorithms as to which prospects you would like to buy to be able to email to. The prospect list vendors will want to sell you the most number of prospects. You, on the other hand, want to gain the greatest return for the least investment so you need to establish the “cut off point” for the prospects you are willing to pay for.

This area of vendor prospect lists is just one area where Excel and Access do not easily “make the grade” of analysing the data that is readily available, for a price, to determine if it is worth that price. It is also a good example where you will be rightly told “no” you can not have it in the data warehouse until you have established the value of the lists.

This is an area that falls in “the middle bit”. An area where you need a more robust, more agile, more scalable suite of tools to establish the value of the data that has now become available.

In some cases? The value of the data will not justify putting the data into the data warehouse and yet there is still enough value in the data to make it worthwhile keeping. So what can you do then?

With the unique Meta5 Capsule environment you can productionize the applications you built to test the value of the data and you can process it yourself on an ongoing basis until such time as it is valuable enough to incur the cost of putting it in to the data warehouse. Or, as the case my happen over time, it loses it’s value and it is no longer worth buying this data from the list vendor.

Of course, this is just one small example. There are now so many data vendors out there providing all sorts of valuable and useful data that you, the Power Data Analyst have really got your work cut out for you to review, analyse, assess and recommend which such vendors you might like to engage and purchase their data.

Meta5 will make such difficult work routine.

Meta5: The Better Way
Thank you for your time and attention.