Kafka Connect is a distributed data integration framework within the Apache Kafka ecosystem. It establishes reliable data flows between Kafka and other systems. A Connector, the core component of the Connect framework, encapsulates the logic for interacting with external systems and provides standardized data read and write capabilities.
The TDengine Kafka Connector writes Kafka data into TDengine TSDB. It reads messages from Kafka Topics, parses the data according to configuration rules, and writes it in batches to the target storage. Batch writing delivers substantially higher throughput and lower resource consumption than writing records one at a time. The connector runs in distributed mode with horizontal scaling: you can adjust the number of Worker nodes to match your data volume.
Installation and configuration
Before deployment, confirm that both the Kafka cluster and the TDengine environment are ready. The connector ships as a JAR file, deployed onto Kafka Connect Worker nodes.
A typical configuration looks like this:
| Parameter | Value | Purpose |
|---|---|---|
| name | tdengine-sink-connector | Connector instance name |
| connector.class | com.taosdata.kafka.TDengineSinkConnector | The connector implementation class |
| tasks.max | 4 | Number of concurrent tasks |
| topics | mytopic | Kafka Topics to subscribe to |
| url | jdbc:TAOS://localhost:6030 | TDengine JDBC connection URL |
| username | root | Database user |
| password | taosdata | Database password |
| database | test | Target database |
The name parameter identifies the connector instance within the Connect cluster. The connector.class tells Kafka Connect which connector implementation to load. tasks.max defines how many parallel write tasks run simultaneously. topics specifies which Kafka Topics the connector reads from.
Data mapping and transformation
Kafka messages commonly use JSON, Avro, or Protobuf formats. The TDengine Kafka Connector supports flexible data mapping: you define the target table schema and Tag fields based on the structure of incoming Kafka messages. Timestamp fields map to the primary time key, numeric fields become measurements, and string fields serve as Tags.
The connector also handles transformation rules. It can convert timestamp formats between different representations, apply numeric unit conversions, and extract nested fields from complex message structures. These capabilities reduce the need for external ETL tools upstream of the connector.
Performance tuning
Three parameters have the largest impact on write throughput.
Batch size and linger time. The batch.size and linger.ms settings together determine the granularity of writes. Larger batches improve throughput by amortizing per-write overhead, but they also increase end-to-end latency. Choose values that balance your throughput targets against the freshness requirements of downstream consumers.
Concurrency. The tasks.max setting controls how many write tasks run in parallel. Start with a small number and increase it gradually while monitoring throughput and resource usage. The optimal value depends on the number of Topic partitions, the Write-Ahead Log (WAL) throughput of the TDengine cluster, and available CPU on the Worker nodes.
Fault tolerance. Kafka Connect includes built-in retry logic for failed tasks. Enable the dead letter queue to route messages that fail parsing to a dedicated error Topic. This keeps the main write pipeline moving while preserving the failed messages for later investigation and reprocessing.
Monitoring and operations
Kafka Connect exposes a REST API for querying connector status, task distribution, and processing progress. Monitor three key metrics: message processing latency, write success rate, and error rate. Feed these into your existing monitoring stack and configure alerts to trigger when any metric deviates from its normal range.
Check connector logs regularly for signs of performance degradation. Look for increased garbage collection pauses on Worker nodes, backpressure from TDengine indicating the write rate exceeds what the database can ingest, and uneven task distribution that leaves some Workers idle while others are saturated.
Summary
The Kafka Connector creates a reliable data pipeline from Kafka to a time-series database. It gives teams a standard, configurable way to build stream-and-batch data architectures for IoT monitoring, financial tick data, industrial sensor networks, and other high-volume time-series workloads. With proper tuning of batch size, concurrency, and monitoring, the connector handles sustained write throughput while keeping operations visible and recoverable.
