Shadow reporting is a given, and a process to integrate is needed. Data Warehouse Automation may be the means by which.

The purpose of Business Intelligence is simple

we will likely be looking to answer two simple questions:

  1. Are we doing things right?
  2. Are we doing the right things?

Other questions are simply variations to the above:

  • Are we beating the the competition?
  • Why are we loosing deals in a specific market segment?
  • What should we be doing differently?
  • Why is one project team capable of delivering on time,
    what can the others learn?
  • Am I scoring as expected?
    Or: if we do 5% more of A, and spend less time on B, in the coming 12 months,
    will I get my bonus?

These questions are fundamental, pertinent and pretty simple to ask, so we would expect to get fast answers.
Except that we don't.

On top of that, as we gain experience we will ask more questions as well as look for more profound answers.

Along the way, we will likely discover others in the organisation may have similar questions. So we need to figure out how to share findings. Soon enough we find ourselves needing single version of facts & common definition of factors & dimensions ... hence the birth of a Data Warehouse.

And you will need a Data Warehouse - no question about it.

We will need to build a Decision Support System, will need to manage data quality, need to be able to validate figures, to author, document and verify - while being able to serve many people with many needs. The combination of people and system then rapidly grows to become a Decision Support Ecosystem.

As we proceed, our quest for truth, confirmation and overall improvement will become harder.
The success of the effort becomes the very reason for slowing down.

However, nothing has changed, really - all of us will still be looking for basic answers to simple questions.

Shadow Reporting

In every organisation I have had the pleasure of working with, we have encountered (or been part of) Shadow Reporting. It is often the result of a process, which starts out in the shadow of the private workplace. Shadow reporting flourishes equally well within the confines of department. But, eventually, when proven successful - it will need to get integrated in the mainstream Decision Support Ecosystem. Which might go something like this:

From the shade: shadow reporting 101

  • Being eager to get results, Business users work with material available to them: company reports, department stats, website information, outside material, etc.
  • Then, they try and simulate numbers/missing information. On paper, copy/paste into Excel, growing a departmental database - whatever skill allows them to.
  • Later, we may ask someone for input, like a colleague: "Hey Adam, could you send me a copy of your Excel report from last meeting?"
  • Shadow reporting environment is born

As we proceed we often find ourselves looking for more and more outside help & connection

Stepping into the light:

  • Soon enough we get noticed and integrated into the mainstream ecosystem.
  • Here we find ourselves needing to interact with an ever growing group of stakeholders: the internal IT team, external BI consultants, other departments, overseas colleagues, ...
  • An Enterprise data warehouse is born
  • Derivatives such as corporate reporting and departmental data marts are used to streamline Decision Support processes

As we proceed, we are becoming part of the Decision Support Ecosystem. Additional efforts tend to become more difficult as many more people are involved. Time to market is going up. A lot like being in a traffic jam.

Then, one day, in some office, a young, eager-to-prove Business Analyst will get dissatisfied with the pace. She may opt for a shorter time-to-result and try things on her own. In turn, drawing from material available and within their technical skill, and undergo a similar process.

Another shadow reporting silo is born, and the cycle starts over.

So what's the point?

The point of the story is this: our need for confirmation, improvement and fact-based decision making may be universal.
... but we do not like to be kept waiting.

Based on our need for speed, there will always be room for some form of Shadow reporting.

When successful, the shadow reporting will get noticed, picked up and integrated into the light of the Enterprise Decision Support Ecosystem.*

If we believe this to be a given, than we may turn to the more interesting governance related questions:

  • How can we facilitate this process, i.e. how can we cut time to market?
  • How can we drive most value from individual efforts and shadow reporting projects?
  • How can we facilitate the integration of fascinating, insightful shadow reporting exercises
    and allow for all stakeholders to benefit?

Individual initiatives should be applauded, not halted - but the fruit of the labour should be made available to all, at the fastest possible time to result.

And that's where Data Warehouse Automation has a key role to play.

* This is what some people might refer to as the process of Data Discovery