Building a Virtual Power Plant Management System: Pain Points & Best Practices

Introduction

In the construction of Amber Optimal’s Virtual Power Plant (VPP) Operations Management Platform, the initial estimate involved around 10,000 devices with a total data acquisition volume reaching hundreds of thousands. One of the key concerns in database selection was ensuring the system could handle concurrent data processing for at least 2,000 devices. This article shares Amber Optimal’s database selection experience and the results of applying TDengine, offering valuable insights for others.

Amber Optimal’s Virtual Power Plant Operations Management Platform is a cloud-based product that integrates business processes, automated operations, and intelligent decision-making. The platform allows load aggregators and resource owners to quickly build virtual power plants and optimize the economic value of flexible and adjustable resources through participation in electricity market trading. The product is divided into two versions: the Aggregator Operations Management Platform and the User-Side Management Platform. The Aggregator Platform mainly manages northbound integration with power dispatching departments and electricity trading centers, while the User-Side Platform focuses on southbound integration with various energy resource systems on the user side, including resource assessment, access, and dispatch response.

Challenges in the Project

One of the biggest concerns during the project was the database’s read-write capabilities. With around 10,000 devices involved and the expected data acquisition volume reaching hundreds of thousands, it was crucial to ensure that the database could handle concurrent data processing from at least 2,000 devices. These devices include meters, grid connection points, photovoltaic systems, energy storage devices, charging stations, cooling systems, and more.

To meet the demands of the trading center, the data collection granularity had to be minute-level, and in certain cases, second- or millisecond-level data was required for fault analysis. This put significant pressure on the database’s ingestion performance. Additionally, when responding to virtual power plant invitations or performing load and responsive capacity analyses, large amounts of historical data had to be queried, placing high demands on the database’s query responsiveness.

Currently, TDengine supports Amber Optimal’s Virtual Power Plant Operations Management Platform, providing clear advantages in data storage. The overall compression ratio is 7-8 times, and data queries can achieve second- or millisecond-level response times, offering strong support for algorithm analysis and the data center platform.

Project Deployment Overview

Amber Optimal deployed their project with TDengine using a 3-node configuration, each consisting of 2 cores, 4GB of memory, and 600GB mechanical hard drives. The architecture is as follows:

Amber Optimal chose TDengine as the time-series database (TSDB) for the Virtual Power Plant Operations Management System due to several key considerations:

1. Business Compatibility: TDengine’s product positioning aligns well with the characteristics of IoT and energy-linked dispatch and trading in the virtual power plant sector. Given the high number of industry-specific cases involving energy, IoT, and connected cars sectors, TDengine was a strong fit to support the company’s future business development.
2. Strong Read-Write Capability: Whether it’s data from ten years ago or one second ago, TDengine allows fast querying within any specified time range. During testing, TDengine was able to handle large-scale data retrievals with second- or millisecond-level response times. Data can also be aggregated across time or multiple devices, making it easier to calculate data in various dimensions. During ingestion, TDengine easily supports millions of rows per second.
3. Data Compression: Thanks to its columnar storage, TDengine offers a high compression ratio, which helps save on server disk costs. In actual use, the compression ratio can reach up to 10%, reducing storage requirements significantly.

TDengine Deployment in the Project

In practice, TDengine handles the following modules:

1. Storing Raw Data Collected from Devices: Whether the data granularity is minute-level, second-level, or millisecond-level, TDengine stores the raw data collected from devices in real-time.
2. Responding to Power Dispatch and Trading Centers: TDengine quickly returns the most up-to-date device status data at the millisecond level (using select last_row query) and historical data within a 3-day range (using select * query).
3. Supporting Algorithm-Based Analysis: Using distributed task scheduling architecture (Celery), Amber Optimal’s algorithms like cluster analysis and neural network predictions read raw data from TDengine. These algorithms calculate electrical load curves at 5-minute and 15-minute intervals, predict the future 7-day responsive capacity data, and calculate optimal resource allocation ratios.

Future Plans

In the future, Amber Optimal plan to deploy multiple single-node TDengine instances at each Virtual Power Plant Operations Management Platform on the user side. These instances will not only collect and forward data but also handle time-series data quality governance and real-time model predictions. On the load aggregator side, they aim to build more complex computational metrics and advanced models using TDengine, integrating it with task scheduling, resource activation, demand response, electricity trading engines, and virtual power plant industry standards.