Skip to main content

About the report

During 16 August – 2 September 2021, ESCAP Statistics Division organized the Expert Group Meeting on Big Data for Official Statistics: Data Governance and Partnership Models. The event was aimed at the following – exploring interest and need for regional guidance to NSOs on big data governance and partnerships; exploring institutional and legal foundations for successful big data integration into official statistics; and country experience sharing. The EGM focused on two topics: Big Data Governance and Big Data Partnership models and was comprised of four sessions: two public Stats Cafes and two closed expert discussions - one dedicated to each topic.

The two Stats Cafes on Data Governance and Big Data Partnership Models were attended by over 150 online participants each. The presentations and recordings of the Stats Cafes are publicly available and fed into the succeeding expert discussions. The two expert discussions were attended by 30 participants from the national statistical offices, development partners and private sector.

This report highlights the main points presented and discussed during the four sessions of the Expert Group Meeting

I. National Statistical Regulatory Framework

The National Data Ecosystem goes beyond the National Statistical System (as seen in Figure 1) and comprises privacy and data protection and access to public sector information and archiving acts. These acts guide use and reuse of data at the national level. A national legal framework lays out the rules and data sources for the production of official statistics. Furthermore, regional and global principles, guidelines and frameworks provide additional guidance to national statistical offices.

Figure 1. National Data Ecosystem

Figure 1. National Data Ecosystem

Source: Gabriel Gamez, Principles, Codes and Normative Frameworks, Stats Café presentation on 16 August 2021

In many countries, the statistical legislation is outdated and does not support the production of official statistics with alternative data sources, despite strong pressure to produce more granular and timely data. In countries with capacity and resource constraints, some NSOs are pressured to use alternative sources of data to produce official statistics. In others, there is a pronounced lack of political will from the leaders to use secondary data sources in decision-making. Hence, countries have varying national regulatory framework and experience with integrating alternative sources of data into official statistics.


Few countries manage the overall access to non-traditional data sources through national legislation. Others try to regulate access and use of individual data sources separately. However, the use of big data for public good could also be considered beyond official statistics; it could be considered for improving public services, research and other areas. Ideally, governments and NSOs should work towards a one consolidated data strategy that would guide data collection, use and reuse across both the government and with non-government actors.

Figure 2. Our role in the DEA
Figure 2. Our role in the DEA Figure 2. Our role in the DEA

Source: Simon Whitworth, the ONS’ role in the Digital Economy Act, Stats Café presentation on 16 August 2021

Confidentiality and personal data protection were identified as the main regulatory constraints to using big data for official statistics (Figure 3). The regulatory framework on personal data protection varies across the region. Countries like Indonesia are approving the national Privacy Act in line with the General Data Protection Regulation (GDPR) provisions. Others follow the statistical legislation or the fundamental principles of official statistics, that guide the preservation of personal data confidentiality.

However, when it comes to data sharing across different government entities, the current individual ministerial legislation may impede data sharing because of privacy concerns. This is also true in multiple countries, when attempts are made to access data through the regulator. There are cases where the regulator collects data from the private sector but cannot then share it with NSOs or other public institutions due to legal constraints.

Figure 3. Opinions received during the Expert Discussion on Big Data Governance, 19 August 2021

Figure 3. Opinions received during the Expert Discussion on Big Data Governance,  19 August 2021


Some countries in the region are fine-tuning their statistical acts and need support in identifying best practices of integrating big data into the production of official statistics. One of the questions is how the national statistics offices can ensure that the statistical act has an upper hand over institutional legislation protecting data sharing.

Continuous compliance with the ethics principles and legislation is of major importance to maintain public trust of the statistical office and the statistical system when big data is used. When conducting pilots and data integration, NSOs should continuously show that they are working in the spirit of the legislation and are abiding to the principles of privacy and ethics. The public should understand how their data are used. However, it was noted that ensuring transparency and accessibility of data requires significant time and resources.

II. Scaling up big data projects

The transition from big data experiments to big data use for regular statistical production should occur following the piloting phase to maximize the degree of innovation of the respective application. The NSOs should start by testing many options in a “fail-early” agile way and then build a use case and business case for pursuing the access and use of certain data for a specific application. There are several benefits to such an approach. These include having a small team of methodologists to carry out the pilot, low costs and short timing needed to assess the viability and needs for further data integration, when assessing the transition requirements in terms of data, infrastructure and resources.

While in most cases alternative sources complement traditional sources of data, there are cases where they can replace traditional production of certain statistics. These cases include using Mobile Positioning Data as a replacement for household survey data to assess domestic tourism in Indonesia, at a third of the household survey cost and with greater granularity and periodicity.

Demand for timely data to inform policy and address public issues can facilitate or speed up big data adoption and integration. For example, in Mexico, which is experiencing fast urbanization, but the census is conducted every 10 years, National Institute of Statistics and Geography (INEGI) is monitoring city expansion through satellite imagery. Similarly, in Australia, satellite data is used for dwelling assessments, saving money and time. Therefore, building a business case for data use to solve a pressing issue could facilitate faster access to data along with important cost savings.

III. Big Data Partnership Models

To use alternative sources of data for official statistics, national statistical offices should secure access to those data first. These data can come from both within the government and non-government institutions in the private sector. New models which move beyond the traditional philanthropic initiatives are emerging, as highlighted in Figure 4.

Figure 4. Emerging models for NSOs access to MNOs data

Figure 4. Emerging models for NSOs access to MNOs data

Source: Martina Barbero, Stats Café presentation on 30 August 2021


COMMERCIAL MODEL. NSOs can access private sector data through commercial models, by paying companies full or preferential price for accessing data or services.

RECIPROCITY MODEL. Another model is the reciprocity model, where the NSOs identify non-monetary incentives for the private sector, such as producing aggregated data that would allow the company to calculate its market share.

GOVERNMENT LEGISLATION is an approach taken by several countries in Europe. This is an example of UK’s Digital Economy Act and the European Data Act.

INTERMEDIATION BY THE REGULATOR. Another approach discussed was the intermediation by the regulator. However, several NSOs pointed to a challenging relationship with the regulator, reiterating that the regulator may not share data with the NSO due to limiting regulatory provisions.

OTHER MODELS. Also, there are cases where it is not useful to involve the regulator in big data projects. An example is collaboration of the NSO with commercial banks. The Central Bank, as the regulator, may be one of the consumers of the insights, which could make data suppliers reluctant to share data. In such case, the NSO could leverage its independence and not involve the regulator.

On the other hand, there are cases where it is useful to get other government departments or regulators involved for strengthening the use case for sourcing data. Moreover, there are cases where the private sector may see the benefit of the regulator getting more involved.

There was another partnership approach highlighted by ONS. Facebook does not share data with the government but does so with universities for research purposes. Hence, ONS partnered with universities for aggregated insights based on Facebook data.

Through most of these partnership models NSOs do not gain access to raw data, but rather to insights and new services provided by the private sector.  When using big data, NSOs do not have control over the data nor the ways those data are collected. For administrative data, public authorities must inform the NSO in advance about any changes in the data they collect or the ways they collect them. It is much more difficult to stay informed about any data-related changes from the private sector. Access to and sustainability of the private sector data are uncertain over the long term. As NSOs want to be in control of the data they produce, they are more reluctant to deal with new data sources that are highly volatile.

Whatever the collaboration model, the NSOs need the following:

  • to have a clear understanding of the use case of the data they are requesting access to,
  • to understand the organizations, they are about to approach,
  • to explore win-win collaboration models and communicate them efficiently to their partners,
  • to have the right technologies, skills and legal support,
  • to have supporting legislation and/or enter in MoU agreements with data providers,
  • to be transparent in the way they use citizens’ data,
  • to be good facilitators and moderators.

IV. Challenges

National statistical offices face multiple challenges when considering big data use for official statistics and when entering big data partnerships.

The following issues were discussed during the Expert Group Meeting:

Legislation and privacy protection

Legislation is the starting point for the use of big data for official statistics. Countries need to improve their statistical legislation to allow for using alternative sources of data for official statistics and to regulate access to private sector data. Since the national regulatory frameworks differ across countries in the region, NSOs are exploring the most suitable partnership models. NSOs may also get more involved in the drafting and consultations of other big data, data sharing, digital government and related legislation, to ensure the coordination of cross-government data efforts.

Personal data protection and privacy remain of a very high concern and needs to be addressed at the national level. It was also noted during the big data partnership models discussions, that NSOs should not give the impression to citizens that the new data partnerships would relax data privacy rules.

Procurement, people and skills

Most NSOs are not set up to tackle data partnerships for new operating models and do not have expertise to negotiate new partnerships with the private sector. The NSOs lack experts in negotiations with the private sector and at the procurement stage and are often asking statisticians and data analysts to become procurement specialists. The NSOs usually rely on a small legal team to help with all their needs, which means that lawyers are not sufficiently equipped and staffed to deal with unfamiliar partnership models and legal agreements.

Under the current conditions, the NSOs cannot operate at the expected pace and procurement requirements, and the resources dedicated to it need to change to respond to new demands.

Data access costs

There are many debates around the access costs for private sector data. Some argue it should be provided for free as the official statistics are a public good. Others recognize the costs that private sector incur to provide their data and services to the NSOs, and hence justify paying marginal costs. Other NSOs may choose to pay market or preferential price, but sustainability of long-term access usually remains uncertain for such models. Also, some NSOs may not want to set a precedent for commercial model with other vendors. However, by choosing this approach, they may not be able to get any data at all.

Furthermore, not only the costs of accessing the data should be considered, but also costs incurred due to reverting to traditional data collection systems and the opportunities lost when traditional data are not available.

V. Evolving role of the NSO in the era of big data

The role of the NSO is evolving in the big data era, but the pace of change varies across countries. NSOs are constantly under growing demand for timely and disaggregated data.

The use of big data calls for modernization of the NSOs, which sometimes lack self-confidence and are afraid to engage in partnerships with data holders. Big data cannot be fed into existing processes, and its adoption requires changes in the business processes. Moreover, changing the business model is as important as adopting the business process. Also, it is easier to adopt big data where there is less of an established structure. It is important to note that being equipped for big data depends on factors such as the context under which the NSOs operate and what percentage of the population has access to Internet, as new data must go hand in hand with social development.

One of the NSOs’ role is to reduce duplications on data demands, and they could also play a more important role in the national data ecosystems and the supporting regulatory framework. Some NSOs are assuming the data stewardship role or adjusting their current role to meet the demands of big data, which calls for changes in the statistical organization (Figure 5). In this role, NSOs need to build trust and joint scope with partners on what needs to be achieved.

Figure 5. From Authority to Stewardship

Figure 5. From Authority to Stewardship

Source: Gabriel Gamez, Principles, Codes and Normative Frameworks, Stats Café presentation on 16 August 2021


Many current business processes in the NSOs are not adapted to data integration, as many departments are working in silos. The transition to big data requires changes in the organizational structure and either employing more staff or changing the way current staff work. NSOs should not only hire data scientists, but pull the necessary skills already present within the organization and have people collaborate more, both internally and with other organizations both public and private. Also, the NSOs should create a safe space for innovation, thus transitioning into learning and dynamic institutions.

VI. Next Steps

Reflecting on key take-aways from the event and the next steps for future collaboration on big data for official statistics, participants highlighted legislation, knowledge sharing and training as priority areas (Figure 6). The discussions provided a better understanding into the NSOs’ needs and areas where development partners could better support them.

Figure 6. Answers received during the Expert Discussion on Big Data Governance, 2 September 2021

Figure 6. Answers received,  2 September 2021


NSOs need guidance on accessing data sources, but also to justify their work to government and the parliament. As many NSOs are fine-tuning their statistical legislation or exploring uses of alternative sources of data for official statistics, several areas of support have been identified.

NSOs need guidance on accessing data sources, but also to justify their work to government and the parliament. As many NSOs are fine-tuning their statistical legislation or exploring uses of alternative sources of data for official statistics, several areas of support have been identified.

  1. NSOs need international guidance on the main regulatory provisions conducive to big data use for official statistics, along with checkpoints that could give the NSOs room to slow down and focus on priority developments. It was noted that the country regulatory context is overestimated. There are similarities in the countries’ approaches with limited country specific context. ESCAP could distil the common threads for replication studies. NSOs also need support through examples of legal frameworks conducive to big data use for official statistics and privacy protection. The NSOs are interested in the necessary revisions to the Statistical Act that would give them an upper hand over institutional legislation protecting data sharing and guidance on accessing data from the private sector.
  2. The NSOs need access to big data related best practices to better understand the big data landscape, uses cases, partnership models and legal frameworks. There is a need for a system of detecting and documenting good examples as they emerge across countries in the region and beyond. While NSOs need to develop data partnerships within the country, they should also learn from and develop partnerships with other NSOs. They would appreciate learning opportunities from other NSOs.
  3. Regional and global organizations could negotiate data access from multinational companies at regional and/or global levels to benefit from attain economies of scale. They could also provide guidance on the use of regional big data hubs and platforms.

Statistics Division +66 2288 1234 [email protected]