Business Intelligence

Business Intelligence (BI) has been around in one form or another for many years and has developed from printed reports or a weekly statistics email to interactive dashboards.

Despite the availability of different BI tool and the creation of internal BI teams, this area remains complex, with many organisations still dissatisfied with their BI and many BI teams struggle to adopt a vision for business intelligence that gets strong internal support.  

Business intelligence strategy

Delivering effective BI means lifting above the operational view and taking a more strategic approach. 

1. Ensure basic data literacy

Organisations will benefit from a strategic decision to coach, train and develop as much of the staff as possible, so that they can become data-literate and contribute to the overall business intelligence ecosystem within the organisation. To understand why this is a critical, see our notes on Evidence-Led Organisations.  

2. Support power users

There have always been staff with a high-degree of data literacy who have used basic tools such as spreadsheet software to enhance their work.  An organisation can derive significant benefit from providing these people personal BI tools and access to data so that they can explore and analyse data themselves and so generate useful BI.

This is often the source of internal friction as the tools that BI teams use are generally very expensive and so not suitable for a widespread rollout.  A number of vendors of high-end commercial tools are recognising this and have begun to provide free or low-cost personal tools to fill this gap.  To be clear, this means dashboard/visualisation creation tools, not viewing tools which are much cheaper but do not give people the ability to create their own BI.  

3. Transform the role of the BI team

Adopting this strategy is significantly easier if the BI team transforms to take on the following responsibilities:

  • A centre of excellence for BI skills, mentoring and training staff to help them become data-literate and proficient in producing their own business intelligence.
  • Support centre for the personal BI tools used by staff.
  • Publishing data catalogues and making data accessible to the rest of the organisation. 
  • Engaging across the organisation to identify what datasets people would use and then procuring and publishing those.
  • Opening up central dashboard or reporting systems so that all staff can self-publish the BI they generate.

A BI team is often well suited to take on key aspects of the data governance function, which is responsible for setting the overall framework for the management of data. 

Engaging BI

Data visualisation

Data visualisation allows people to gain insight from data quicker than looking at the raw data.  In less than ten years advanced data visualisation has gone from niche to mainstream and there are now many tools that support a wide range of useful and innovative visualisation types.  Rather than being seen as a “nice to have”, data visualisation should be considered a key output of business intelligence.

Dashboards

Dashboards bring together a set of visualisations to give a broad view of a subject and are increasingly important as this reflects the broadness of strategic goals and targets.  Interactive dashboards have the most impact as they allow people to conduct limited exploration of the data and produce greater insight.  Interactive dashboards should not be seen as a “nice to have” but a key output of business intelligence.

Managed discussion

Business intelligence can be made much more valuable by enabling managed discussion around specific data points or charts.  For example, if someone is able to annotate a chart and write “does anyone know why this is trending down?” then the subsequent discussion will normally generate insight that has so far been missed, or only shared by a few people. 

Data repositories

Business intelligence relies on good access to data, which is generally held in a variety of systems throughout the organisation.  The choice of mechanisms and tools for providing that access is a complex one, with multiple factors to consider. 

Problems

There are some common problems the BI practitioners face when accessing data:

  • Access.  Some systems can be hard to query and extract data from.
  • Structure. The way the data is held in some systems makes it difficult to process or use.
  • Efficiency. Even simple queries can sometimes require the extraction of huge volumes of data from a system, which is inefficient if the same data is repeatedly extracted.
  • Interoperability.  Different systems refer to the same data in different ways making it hard to create queries that combine data across systems.
  • Performance.  Directly or repeatedly querying a source system may add sufficient load to impact its operational performance.

Data warehouses

To address these problems, Data Warehouse products appeared.  These are highly structured data stores where the data that is ingested is shaped and formatted into a consistent form, solving many of the problems above.  Unfortunately, data warehouses have their own problems, namely that it is too much work to structure the data from each new system and so data warehouses rarely contain all of the data that people need access to.  The risk is that a data warehouse only contains the data that is easy to work with and so cannot support much of the analysis people want. 

Data lakes

The solution was a new breed of Data Lake products, which take the data from systems as is, allowing far more data to be ingested, and then create the structure and interoperability at the time of analysis.  However, data lakes also have their problems, including lots of data being ingested but never used and slow processing as the data is structured for each query.  The risk is that a data lake becomes a data swamp, full of data that nobody ever queries because it is too hard.

Streaming, real-time, near real-time and batch processing

Another complication with data repositories is what forms of processing do they support:

  • Streaming.  The data is inspected as it is generated, generally to identify issues or anomalies that are then flagged for later analysis. Once processed the data is either discarded or a short rolling window is kept, which is sometimes described by the labels “real-time” and “near real-time”
    • Batch. This is scheduled or on-demand processing of a full dataset.  The requirement may still be for a fast response to any query.

Where to from here

Many of the problems identified above are being solved as applications mature, making it easier to work directly with source systems and thereby enabling hybrid products that both query source systems directly and pull data into a repository as needed.  These developments are largely coming from the addition of REST APIs as a means of extracting data from systems:

  • Access.  Increasing use of APIs.
  • Structure.  As part of the adoption of REST, the data provided is getting simpler and with machine readable structures.
  • Efficiency.  Through provision of an API call that allows only data that has been added or changed to be retrieved.
  • Interoperability.  Again, the adoption of REST is helping to create common structures and so enhance interoperability.

Internally developed systems

It should be clear that internally developed systems need to have REST APIs for access to their data if they are to be usable by the broadest possible range of BI and analytics tools.

How we can help you with Business Intelligence

  • Developing a data-literacy training programme for your team
  • Advising on business intelligence tools and online services
  • Advising on the architecture of a data repository and how systems feed into it
  • Review an existing business intelligence team or strategy and recommend improvements