Data ingestion in industrial internet and IoT projects comes with real pain points. Enterprises must connect to diverse heterogeneous data sources: PLC sensors, MQTT device gateways, Kafka message queues, OPC UA servers, and more. Each protocol has its own SDK and development conventions, forcing development teams to write dedicated collection programs for every data source. The complexity of data ingestion directly affects project delivery timelines and ongoing operations costs. This article explains how TDengine replaces traditional development approaches with a configuration-driven model, enabling efficient ingestion across multiple protocols.
The pain points of traditional data ingestion
Three core pain points: Heavy development workload (a mid-size project involves 5 to 10 data source protocols, and connecting a single data source can take hundreds of lines of code). High operational complexity (each custom collection program needs separate monitoring for runtime status, exception reconnection handling, and log management). Inconsistent data models (different data sources use varying field names, data types, and timestamp formats, requiring manual mapping logic for each one).
The core of zero-code ingestion: taosAdapter
TDengine achieves zero-code data ingestion through the taosAdapter component, a lightweight data ingestion gateway. It has built-in support for multiple mainstream industrial and IoT protocols. Three core responsibilities: protocol adaptation, data model mapping (automatically mapping external data points to Supertable and Subtable structures), and concurrent write optimization (with built-in batch aggregation and write buffering).
Supported protocols and data sources
MQTT protocol: Built-in MQTT client that subscribes to specified Topics. Configuration requires only the Broker address, Topic name, and target Supertable. Kafka protocol: Acts as a Consumer that consumes Topic messages, supporting JSON, InfluxDB line protocol, and other formats. OPC UA protocol: Connects to OPC UA servers and subscribes to node data changes. PI System migration: Provides migration tools and protocol compatibility solutions, supporting both batch historical data migration and real-time data synchronization. REST API and InfluxDB compatibility: Provides RESTful write endpoints and InfluxDB line protocol compatible interfaces.
Configuration methods and data model mapping
Two configuration methods: Configuration file approach (define connection parameters for each data source; once started, connections are established automatically and data collection begins). Management console approach (a visual management console that supports adding, editing, and monitoring data sources through a web interface, displaying connection status, write throughput, and error logs in real time). Data model mapping: Users create Supertables in advance to define the field structure. taosAdapter automatically creates a corresponding Subtable for each device and maps external data fields one-to-one with Supertable columns.
Typical application scenarios
Industrial OPC UA data collection: A large manufacturing enterprise needed to collect real-time data from hundreds of PLC devices. Operations staff configured the OPC UA server address and node list through the management console, reducing deployment from weeks to hours. MQTT IoT device ingestion: A smart campus connected over a thousand environmental sensors. By configuring the mapping between MQTT Topics and Supertables, all device data was automatically ingested into the database.
Comparison with traditional ETL solutions
| Dimension | Traditional ETL Development | Zero-Code Ingestion |
|---|---|---|
| Development cycle | 2 to 4 weeks | 2 to 4 hours |
| Code volume | Hundreds of lines per data source | Zero code |
| Operational complexity | High, multiple collection programs to maintain | Low, unified gateway management |
| Protocol extension | Requires new adapter module development | Configuration-based support |
| Data model consistency | Depends on development standards | Guaranteed by automatic mapping |
Conclusion
Data ingestion is the first step in deploying a time-series database, and it is the step most easily underestimated. TDengine encapsulates protocol parsing, data transformation, and model mapping into configuration-driven operations through the taosAdapter gateway, lowering the barrier for ingesting multi-source heterogeneous data.
