Chapter 3:
Pioneers of data transparency
In this chapter
Economies demonstrating good practice
This section sets out to understand open data practices in economies that had excellent scores. New Zealand, Australia, and India had excellent scores on data transparency, with many others economies scoring excellent on one or two criteria. There are lessons to be learned from them, so to inspire what good data transparency practice can look like for such a diverse region.
1. Time granularity
Ideally, power data needs to be published at a minimum of hourly intervals for an optimal power market operation and analysis. Every national power grid likely collects some level of data for balancing operations. This data must be open to the public to enable effective decarbonisation. New Zealand, Australia, Japan, Bangladesh, Singapore, Sri Lanka, and Georgia are sharing data at hourly or less intervals. Both Australia and New Zealand went beyond this standard, publishing data through a public dashboard every five minutes, featuring real-time data sharing. This level of granularity allows each economy and its citizens to give more granular price signals for better generation and use of energy, increased ability for balancing of supply and demand through demand response technologies, as well as improved trading incentives. This requires a fast and at times expensive data network, especially for larger nations, and may be difficult to implement for lower-income economies.
2. Publishing lag
Balancing operations and accurate forecasts becomes harder as data is made available weeks, months, and years past the time of generation. In particular, multiple years of time lag diminishes the value of data significantly. India, Australia, and New Zealand all scored excellent by publishing data in real time or the next day. To do this, data administrators need to keep databases updated at all times, requiring constant maintenance and vetting.
3. Geographical granularity
Over half of Asian economies published data nationally, which obscures information on where generation is occurring. Geographical granularity is especially important in large nations like India, where power may travel hundreds of kilometres to its point of consumption. Ideal data transparency goes beyond the regional level, identifying which units inside which power plants are generating electricity. India, Australia and New Zealand all published data at the unit-level.
4. Fuel breakdown
Estimating carbon intensity of power and understanding how the electricity mix needs to change requires detailed information on the fuel source of generation. Only Australia, New Zealand, South Korea, and Taiwan provided detailed fuel breakdown, including off-grid generation. India provided information on all fuels, but reported off-grid generation inconsistently. Australia consistently reported off-grid solar generation, including other fuel breakdowns for gas (Open Cycle Gas Turbine and Combined Cycle Gas Turbine) and coal (black and brown). New Zealand, which has a comparatively higher percentage of active geothermal power plants, also gives an additional breakdown for geothermal energy.
5. Ease of access
The ideal format of data is free, publicly open data that does not restrict overseas access. Data must ideally be accessible via public API for developers to retrieve data programmatically. For non-developers, data must be shared in a machine-readable format, such as xlsx or csv. Australia and New Zealand were the only economies to have functioning public API’s. India has public APIs for some datasets, but not all. Many economies struggle to make data available for many reasons. For one, the government or public utilities may want to retain data to prevent public scrutiny or various forms of intervention. Some economies lack technical and/or financial capacity, and political will in the public sector to enable good data governance.
6. Additional Data
While this study focuses on generation data, other power sector data is desirable. Capacity, demand, wholesale price, tariffs and consumption by sector data are some of the key additional power sector data. India, Australia, New Zealand, and South Korea, Taiwan, and the Philippines are some of the example economies that made every category of additional data mentioned available.
3.1
[Case study] Indian government’s partnership with Vasudha Foundation
This section highlights success stories from the region to serve as inspiration for peer economies. In particular, India is an exceptional case, as it has made significant strides in making its vast power sector data available in one place in partnership with the private sector.
NITI Aayog, a policy think tank of the government of India, has partnered with the Vasudha Foundation, which has become a leading aggregator of energy data across India’s 28 states and eight union territories. Through a cross-sectoral and data-driven analysis as well as creation of comprehensive data repositories, Vasudha Foundation is aiming to create a one-stop open access data platform to present a holistic picture for India’s efforts on climate change and a low carbon pathway.
Public-private Partnership between NITI Aayog and the Vasudha Foundation
1. Key stakeholders
Vasudha Foundation is a non-profit energy and climate policy think tank that tracks and supports India’s clean energy transition through data driven analysis. Vasudha Foundation has developed a comprehensive power sector value chain data comprising electricity generation, transmission, distribution and consumption. Vasudha Foundation has been collecting India’s power plant data to answer the questions of power plant locations, pipeline and retired capacities, what fuels they use, and how often outages happen. Following on from their initial launch in 2021 of Vasudha Power Info Hub, the Vasudha Foundation decided to merge their power data dashboard with the India Energy Dashboard (IED 2.0) developed by NITI Aayog in 2021; a government agency whose mandate is to oversee the adoption and monitoring of Sustainable Development Goals in India. NITI Aayog also shares the key initiative to enable easy and integrated data access for energy and climate. The India Climate and Energy Dashboard (ICED) is a user-friendly platform that aims to bring together comprehensive historical and current data to provide a single window access for all datasets related to climate and energy. It is planned to be launched in the first quarter of the financial year 2024 (https://iced.niti.gov.in/).
2. Solving India’s Power Data Problem
Creating a comprehensive power sector dataset for all of India imposed many challenges. To build the dashboard seen today, the Vasudha team had to collect data from multiple reports and portals, including 400-500 tariff orders and company reports that were scattered across over 68 distribution companies. The reports were only available in pdf format or scanned images, thus making it difficult for the workflows to be automated. However, a public API for automated workflows is in development despite this hurdle.
Vasudha Foundation has developed close relationships with distribution companies and government agencies to gain access to the data used in their dashboard. They have established a high reputation in the power sector, which led to high trust with key data providers. All the power sector data on the portal is from government sources. Their partnership with NITI Aayog has been instrumental for furthering opportunities to develop connections with other ministries and departments to access granular data. Vasudha therefore sees this building of relationships in the energy sector as key to achieving a successful data tool, an insight echoed by Subak’s Power Plants Data Research Initiative.
Through this government partnership, the Vasudha Foundation is seen as authentic and reliable, as their data is consistently backed up by official sources. The Vasudha team remarked that the dashboard now provides a ‘birds-eye’ view for all stakeholders involved in India’s energy transition. Through the creation of this one-stop shop for India’s power data, the Vasudha Foundation has successfully addressed India’s challenges in power data transparency. While it is open for anyone, researchers, campaigners, financial institutions and policymakers have hugely benefited from the tool, as they can easily understand the status quo of India’s journey in clean electricity transition.
3. Technical Data Challenges
Common technical challenges that the Vasudha team faced were different granularities across sources, inconsistent data formats, inconsistent time lag, data accessibility, disagreements in data sources and frequent occurrence of missing or erroneous values. The Vasudha team cross-verified data from multiple sources and identified gaps in datasets.The gaps in data occurred due to reporting errors and mismatch of methodology for the same metric. It was noted that standardisation of metrics, data methodology, granularity and nomenclatures across various distribution companies and government agencies is required to resolve this issue.
4. The way forward
The Vasudha Foundation has plans to collect data on energy use and consumer behaviour. The Vasudha team explained that building demand-side data is a top priority going forward. Without granular demand-side data, it will be difficult to augment the grid according to users’ needs and maintain grid stability as intermittent renewables increase. The urgency of this cannot be overstated, as the energy transition is happening fast not only in India but across Asia.
3.2
Recommendations
Reflecting on the case study, we propose a few low-barrier recommendations from the Open Data Policy Guidelines published by the Open Data Policy Hub (ODPH) as a solution to achieve better data transparency in Asia. We have selected recommendations that can be implemented without a large technical overhead, so that they can be implemented without financial or technical hurdles.
ODPH suggests that 1) data formats should allow for maximal technical access; and 2) providers should publish data in comprehensive and appropriate formats for varied uses. This need is apparent in the Vasudha Foundation’s reporting that pdf or scanned image formats were a barrier to collecting data because they are not machine-readable. Pdf and html documents may be fit for the purpose of delivering analytic content, but are not suitable for sharing data. Therefore, original data tables should be attached in machine-readable formats, including xlsx, csv or json. The Australian Energy Market Operator (AEMO) demonstrates good practice. It provides their quarterly energy dynamics report as a pdf but has an xlsx workbook to support this which contains all the data in the report.
Additional ease of access recommendations include 3) removing restrictions for accessing information. Various sources identified had access limitations such as geography or sector. The goal of open data should be to provide non-discriminatory, free access at any time without restriction or justification for doing so. This also extends to the licence, as data should be 4) explicitly licence free or adopt an open data licence. This maximises the usability of the dataset, and allows multiple users to build off of the data such as anyone using Vasudha’s data. Once made open, the data provider should ensure, 5) permanent, lasting access to the data.
Lack of unique identifiers was one of the main challenges that the Vasudha team faced, as well as the 13 interviewees of Subak’s power plants data research initiative. Unique identifiers provide consistencies when collecting and reporting data for individual power plants and units. The 6) use of unique identifiers is essential for the interoperability of energy data and one of the most pressing challenges for the energy data landscape. Additionally, where established, these identifiers should be non-proprietary and public.
7) Central locations devoted to data publication and policies are essential. Some economies have put efforts to create a one-stop-shop, including India, Australia and New Zealand. However, in many economies, data providers for power are scattered. 8) Digitisation and distribution of archival materials is also essential for these central locations. In many economies, historical power data may need to be digitised. Often, old historical data are archived separately, which makes historical analysis difficult. Azerbaijan demonstrated good practice, whose State Statistical Committee published data from 1913. This further illustrates the need to 9) publish bulk data to enable easy access to all information.
Related Content
