Before I begin, I want to be clear that I’m not talking about AI, machine learning, modeling and the buzzwords that come with them. I’m talking about raw Business Intelligence that provides insights to help businesses make decisions based on evidence and not a hunch.
When looking at data enablement and Business Intelligence, it is easy to go down the rabbit hole and think you need a full machine learning data strategy, but most of the time, all you really want to know is “What is my ROI?” or “What is the impact of email marketing?”. To achieve this, there is a clear data pipeline from data source, to data warehouse, to BI tools. However, the way your organisation manages this process can have a dramatic impact on the value of those insights, and cost of that business function.
ETL - the established method
Traditionally, to gather these insights the process of extracting data, transforming, and storing content in the perfect format ready to present was largely owned by technology teams and data engineers. This is the ETL process: Extract, Transform, Load. The steps are:
- Find insight requirements from business
- Create ingestion process to Extract the data required
- Transform the data into a format that helps represent the report
- Load this into a separate data store
- Attach BI Tool or generate report from this store
This worked great for a long time, and is no doubt still the process for many organisations. However, there are three key factors to note on why this might inhibit the function your setting out to do, why this is changing.
1) Large dependence on technology teams
The function of BI is to get the right insights to the business so they can make considered decisions to help move the business forward. The faster the business retrieves these insights, the faster it can act which can have a dramatic effect on various outcomes.
If new insights are required in the established method, there is an onerous process by involving everyone from Business to BI Analysts to the Technology team to get the data for the insights required. Depending on the size of the organisation and how backlogs are structured for the delivery or data teams supporting the BI function, it could be days or weeks to get the desired insights. To evolve this we need to remove the dependency on the Technology team, and allow the BI analysts work directly with the source data and interact with the Business only.
2) Data is cheap
Because data is transformed, aggregated and manipulated before the Load, it is possible to store much less data on your destination report DB. This may have been an important consideration when data storage was expensive, but times have changed. Every man and his dog now has a data warehouse, and cloud providers have made it cheap and accessible to setup, configure and store huge amounts of data. This means we can just extract all the data and load it into data warehouse where Analysts have access to it. Technology teams can setup Ingestion tools like Azure Data Factory, FiveTran or Stitch can simplify this, and then won’t be beholden to provide any additional transformation, or formatted data, as theoretically the Analysts now have everything at their fingertips.
But how is the transformation and manipulation done?
3) Analysts can be engineers
BI analysts have significantly changed their role over the last decade and are becoming more broadly skilled, knowledgeable and practical in the digital world. They are capable of getting stuck into SQL, doing aggregation and manipulating data to suit their reporting needs. This bridges the gap between a data engineer and an analyst slightly when it comes to querying and manipulation, but there is still a clear separation when it comes to transmission and maintenance. Giving the analysts this transform and manipulation function moves ownership of the Transform task from away from Technology teams and to the BI analyst teams after the Load is complete.
ELT - a small shift for big rewards
With a small shift, we can take advantage of these new developments in the data world. If you let the technology teams facilitate the Extraction process for all data into a data warehouse, without worrying about the Transform, their job is done (apart from maintaining the extraction), and will no longer be a dependent team to facilitate reports. The analysts will be able to pick-up the data stored, merge it into the format they need, and use the BI reporting tools to generate the insights they need.
And if those insights need changing, or there is demand for more, you no longer need to go back to the Technology teams and put requests on their backlog,
- Analysts are unlocked and spend less time conjuring technology requests
- Less resource allocation required from Technology
- Insights can be achieved quicker which allows the business to make decisions faster
Moving ETL to an ELT model is win-win for all. There were logical reasons for the traditional model of enabling BI teams, but now it’s apparent those methods have the potential to create a hindrance for the execution of BI.
If you’re about to undergo a new BI project, or if you’re frustrated with the current function of BI, think about how the teams are structured and make full advantage cloud infrastructure to give your BI Analysts some independence.