DataX Officially Supports TDengine for Better Data Migration

Shuduo Sang
Shuduo Sang
Share on LinkedIn

DataX is an open-source platform for offline data synchronization developed by Alibaba Cloud. It currently supports importing from and exporting to many data sources, including MySQL, Oracle, HDFS, and HBase.

According to its developers, “The DataX framework abstracts a data synchronization workload into a reader plug-in for the data source and a writer plug-in for the target system. This architecture, in theory, can synchronize data from any source, and every new data source that you add can interconnect with your existing data.”

The TDengine team has created the TDengineReader and TDengineWriter plug-ins so that you can use DataX for migrating time-series data to or from TDengine. These plug-ins have now been officially merged into the main branch of DataX, and TDengine is listed as an officially supported time-series database (TSDB).

The TDengineReader plug-in provides the following features for data migration:

  1. Data filtering through SQL
  2. Task partitioning based on time intervals
  3. Support for all TDengine data types
  4. Batch reads, with the size being configurable through the batchSize parameter

The TDengineWriter plug-in provides the following features:

  1. Schemaless writes into TDengine through the OpenTSDB JSON line protocol
  2. Batch writes, with the size being configurable through the batchSize parameter

With the TDengineReader and TDengineWriter extensions, DataX is an excellent solution for importing legacy data into TDengine and exporting data from TDengine.

For more information, see TDengine Migration Tool based on DataX.

For the latest on the implementation of TDengineReader and TDengineWriter in DataX for data migration, check the GitHub repository.

  • Shuduo Sang
    Shuduo Sang

    Shuduo Sang is an open-source software veteran at TDengine, working on magnifying TDengine capabilities to connect easily with more third-party software for IoT and DevOps users. He and his team develop and maintain the multiple client libraries and tools for TDengine such as taosBenchmark and taosAdapter. Before joining TDengine, he worked on open-source and Linux at Canonical and Intel.