What is data assurance?
Assurance of data and data practices is really important for organisations to build trust, decrease risks and maximise opportunities, as we discuss in this blog post. But what does this mean in reality?
At the Open Data Institute (ODI) we define data assurance as:
The process, or set of processes, that increase confidence that data will meet a specific need, and that organisations collecting, accessing, using and sharing data are doing so in trustworthy ways
Dr. Milly Zimeta, ODI Head of Policy, spoke on a CogX panel on 15 June 2021 – about the importance of building trust in data and data practices across businesses and explains data assurance very well in this clip ‘Assuring Data from different perspectives’
Get involved
Demonstrate you're trustworthy with data – subscribe to our tools, events and guides
Complete our survey
We'd love to hear how you and your organisation uses data.
We'll give all participants 20% off the ODI Summit 2021 and will enter you into a prize draw to win a place on our 'Introduction to Data Ethics' course.
Start the 10-minute survey here
How does data assurance increase confidence in data?
We believe the adoption of assurance products and services should both reassure organisations who want to share or reuse data, and support better data governance practices, fostering trust and sustainable behaviour change within those organisations or communities. However, the role of assurance products and services to improve data governance and trust across a data ecosystem are not mature or widely understood.
It is important to look at data assurance from the perspectives of both the data sharer and the data collector or reuser.
Organisations may not be willing to share their data if they are concerned about data quality, how it will be used, its security or what will happen to it after use. Where an organisation wants to limit data use and reuse to a defined purpose, data assurance practices (for example, clear documentation and licensing) can help allay fears of the data being misused, misunderstood or mishandled.
Organisations collecting and reusing data often have to spend a lot of time and effort verifying the quality and completeness of data before it can be used or shared onwards. The same organisation may be concerned about verifying sources of the data, and establishing rights and consent for that data to be used in new ways or shared. Data assurance practices and tools can be used to assess and demonstrate trust in data sources, in various ways for example checking against standards or verifying provenance.
Many organisations collect, use and share different sets of data at the same time and in different ways. At each step in its lifecycle, data has to be tested and verified according to what quality is expected and how it will be used. Data assurance tools and practices can help to provide confidence in that data being fit for purpose, and trustworthy, throughout its lifecycle.
Through our early engagement with ODI members we know that these challenges are clearly understood and being managed in an ad hoc manner, sometimes with considerable gaps in knowledge and resources. The people responsible for data in organisations need access to guidance, advice and tools to help manage data and demonstrate to decision makers and other end users that data can be used with confidence.
[table id=49 /]
Example: A research institute requests patient data from the local health authority in order to conduct research that may lead to an improvement in public health and a reduction in treatment costs. The health authority needs to be confident that the shared data will be used ethically, will not be further distributed and will be kept secure. Providing assurance that the data practices, skills and tools to be used are suitable for this purpose will raise confidence and help encourage data sharing.
Why is data assurance important?
Data assurance supports data governance – the policies, processes, people, standards and technologies that govern access to data assets – and provides more confidence around data flow, fostering trust and encouraging more openness with data. Sharing data in a trustworthy way can create ecosystems where data analysis, insight or tools have significant positive impacts – social, economic, environmental and for public good, and offering benefits for the original data sharing organisation.
The application of data assurance practices is context specific. This means the type of data assurance technique, tool or practice and the extent to which it is applied must be tailored for each context. Here are some examples:
- Where companies want to share product information with their supplier ecosystem in order to improve efficiency and encourage innovation they may choose to focus on ensuring data practices that control access to this potentially commercially confidential are robust.
- Where authorities want to collect and analyse data about the public in order to improve services they may need to focus on reassuring the public that the data about them is being held securely and being managed in ways that protect them from harm.
- In engineering, for example, where sharing component performance data within the sector can improve safety, assurance activities might focus on providing confidence in the accuracy and timeliness of the data.
Data assurance can deliver benefits to organisations, data institutions, governments and authorities in different ways through combinations of:
- Audits of data and data practices, allowing data stewards or publishers to assess their own or reusers' practices, providing confidence that the right people have access to the data and that this process is managed appropriately
- Improving data skills and knowledge of staff and stakeholders, to support innovation with data, legal compliance and drawing insights from data analytics
- Use of standards and technology to access data in different places to gain insight, for example into customers’ behaviour and product popularity or to review the cost and usage of a business’s IT estate to identify redundant systems to decommission
- Transparency and openness, such as through reporting environmental, social and corporate governance (ESG) goals and progress against them
Risks increase without data assurance
We believe that sharing data unlocks social and economic value that can be worth many times any investment. On the other hand, data hoarding can lock in costs and lead to missed opportunities. In many organisations evidence and insight are locked away in unread documents and frozen formats. Data fear springs from concerns around who has access to data and how it might be used. Data fear increases without clear knowledge and understanding. We have developed a model that describes overcoming data hoarding and data fearing.
Without assurance of data and data practices organisations are exposed to the following risks:
- Legal – liabilities are not well-understood or well-managed, potentially leaving data stewards or hoarders exposed to expensive compliance failures
- Financial – inefficiently managed data can lead to higher costs and/or data not being available to users when required
- Reputational – poor security processes leave important data sets vulnerable and hinder data sharing
Example: Trust in an organisation can be reduced when data breaches show that the data is not being held securely. Fines can be levied for data breaches and subsequent reputational damage can lead to a loss of business, and increased barriers and concerns from business partners or customers.
Increasing the maturity and understanding of data assurance
As awareness and understanding of data assurance increases, so too will the expectation that data is part of our national infrastructure and should be accessed, used and shared as widely as possible. This in turn will lead to innovation, and value creation through the development of new products and practices. We believe data assurance processes and tools will increasingly become standard business practice, for example adding data quality metrics and clear statements on reuse to service level agreements. Each step of implementing data assurance practices will help to improve confidence in the quality of datasets and data practices, reduce the costs for processing, foster confidence in sharing and innovative use of data through increased trust.