DataX is an open-source platform for offline data synchronization developed by Alibaba Cloud. It currently supports importing from and exporting to many data sources, including MySQL, Oracle, HDFS, and HBase.
According to its developers, “The DataX framework abstracts a data synchronization workload into a reader plug-in for the data source and a writer plug-in for the target system. This architecture, in theory, can synchronize data from any source, and every new data source that you add can interconnect with your existing data.”
The TDengine team has created the TDengineReader and TDengineWriter plug-ins so that you can use DataX for migrating time-series data to or from TDengine. These plug-ins have now been officially merged into the main branch of DataX, and TDengine is listed as an officially supported time-series database (TSDB).
The TDengineReader plug-in provides the following features for data migration:
- Data filtering through SQL
- Task partitioning based on time intervals
- Support for all TDengine data types
- Batch reads, with the size being configurable through the
batchSizeparameter
The TDengineWriter plug-in provides the following features:
- Schemaless writes into TDengine through the OpenTSDB JSON line protocol
- Batch writes, with the size being configurable through the
batchSizeparameter
With the TDengineReader and TDengineWriter extensions, DataX is an excellent solution for importing legacy data into TDengine and exporting data from TDengine.
For more information, see TDengine Migration Tool based on DataX.
For the latest on the implementation of TDengineReader and TDengineWriter in DataX for data migration, check the GitHub repository.
