Let’s talk about data management, and how an effective data management strategy can give you and your stakeholders confidence in your information.
What is Data Management?
Data management is a process of controlling data that is generated by the operations of your organisation and presenting it in a useful form.
Figure: Data Management Flow
Data management is not a new concept. Prior to modern computing, larger companies had accounting rooms full of people manually processing paper ledgers where they would validate, aggregate, sort, and move data through a sophisticated data management process. In the manual world of information processing any successful enterprize’s understood the need for rigid standards and an efficient process, as mistakes in could involve days or even months of rework.
Move forward to the information age and data volumes that would have previously required thousands of people over many months can now be processed in seconds. This ability to massively scale data management processes has enabled businesses to expand to sizes unheard of in the early 20th century, and to sell products and services that were previously impossible.
The Modern Data Challenges
Modern technology advances have almost all been driven by companies aiming for cheaper, bigger, faster solutions to deliver their strategic goals. This has driven the digital revolution and provided exponentially greater volumes and complexity of data. Many organizations now manage thousands of data interfaces with terabytes (even petabytes) of data.
When it comes to trusting data it’s a misnomer that massive data volumes are the main challenge. It is the lack of understanding in organizational data and the complexity associated with managing it that is the greater the challenge. Organizations often have thousands of systems operating together that are interconnected in a spider web of thousands of real-time integrations and batch processes. On top of the automated processes, is then a wealth of manual processes involving spreadsheets and small databases.
In the 1990’s, data warehouses were touted as the means to better manage and generate insight from organizational information. For most larger organizations these platforms are now a well established part of their ecosystems and are considered critical strategic assets. The problem now for many data warehouses, is as they became critically important, they also stopped being agile and able to adapt to change. This has lead many companies to build multiple data warehouses, or end up with pockets of tactical reporting systems, and a fragmented data management strategy.
As organizations started to be affected by the cost and difficultly of managing data, vendors started to push “easy” pre-packaged data solutions. In almost all circumstances these have caused more problems than they solved, and lead to less understanding of data and in turn more data proliferation. What pre-packaged solutions fail to show is that every organization is different, which unfortunately is not easily packaged.
The Goals
The first part of moving to an organization that trusts it’s data is to set goals and priorities that are not only focused around building bigger, faster and sexier tools. Instead the key to establishing a data management strategy is to start around the old “less is more” principle. With this is mind, then focus on understanding data, removing overly complex and redundant data, and improving information availability both internally and for customers.
Focus on Understanding – In the pre-digital world, data was simple enough that executives tended to have a good understating of their data. In modern systems data generated can be the result of millions of calculations, passing through dozens of disparate applications and manual processes. Being able to verify the accuracy of data and trace it to its origins can be a larger challenge then the initial processing, however it will turn the business focus from “is this data right” to “how do we react to what this information tells us”.
Reduce the Cost of Change – The more data integrations that are put in place, the more difficult can be become to change these. The biggest cost impact if often not the change itself, but the cost to access the impacts of the change on the other systems. This can have a huge impact when large changes are needed, such as mergers, restructures or modernising processes and core systems. Additionally, it hinders the ability to implement small changes, with the spiralling cost of these changes outweighing the benefit. To avoid the cost of change, often tactical decisions are made to create new processes (and not change the old) which in turn causes spiralling complexity and less understanding of systems and data.
Make people accountable for their data – It is always better to have high quality source data, than to try and clean poor quality data later. A common challenge is that the people and groups responsible for the production of source data, are not the ones who directly benefit from it being of a high quality, which unsurprisingly results in poor quality data. The way to address this is to “make people accountable” for their data, and make the accountability real by integrating this into their incentives.
Improve Availability – People who work in large organisations will know the phase “it’s not what you know, it’s who you know” rings true, and most people still gather data by asking the people who have it to provide it rather than having it directly available. Making the data available means that (privacy considerations being accounted for) every employee has access to any information and can understand this information.
Remember the Customers – Data management doesn’t just have an internal impact; it can have a huge impact on the customer, and for most industries, customer expectation is very high.
The Way Forward
The starting point most organizations is to acknowledge that there is no “silver bullet” answer or prepackaged solution that will solve information management challenges. The best way forward is to often take a step backward and instead of implementing new solutions, actually invest in understanding the current data and process.
Some items to consider are:
- Classify data in terms of quality, trust level, currency (i.e. how up to date it is), and sensitivity.
- Trace where data comes from, and map the trust levels against the trust levels of it’s source systems.
- Review usages of data; who uses it, what systems it is delivered to.
As this work start to give insights, then consider where there are “quick win” opportunities that could start to reduced the information complexity. Some of these could be to:
- Remove redundant or duplicated data.
- Improve poor quality data at source.
- Classify data, and educate staff how to use data.
- Apply ownership and accountability to data.
The final part of moving forward is to introduce a longer term information strategy for the organization. The information strategy should lay out the governance and processes will encourage, and where necessary enforce, a data focused culture.
With the understanding in place, and a company culture committed to improving it, the organization is in a great place to get the technical solutions (such as the Information Hub) in place to consolidate and improve data.