High availability is a core requirement for production TSDB deployments. TDengine TSDB supports service continuity through multi-replica mechanisms, automatic failover, and operational controls for recovery and load distribution.
High availability architecture
Architecture components
A typical deployment places a load balancer layer (Nginx/HAProxy) above three taosd nodes (Node1, Node2, Node3), which connect to a multi-replica storage layer.
Replica strategy
Two SQL examples illustrate the setup:
- Create a database with three replicas and 10 vgroups:
CREATE DATABASE demo REPLICA 3 VGROUPS 10; - Check vnode status:
SHOW VNODES;
Multi-replica mechanism
Replica types
TDengine supports three replica synchronization modes:
| Mode | Behavior | Best for |
|---|---|---|
| Strong sync | All replicas must confirm writes before returning | Core business systems |
| Weak sync | Only the primary replica must confirm | Performance-prioritized workloads |
| Async | Primary returns immediately without waiting | Log-type data |
Leader election
TDengine uses the Raft protocol for leader election. Three key properties define this process:
- Automatic election of a new leader when the current leader fails
- Automatic data synchronization across replicas
- Transparency to applications. No application-side changes are required
Automatic failover
Fault detection
Run SHOW DNODES; to check node status. Each dnode reports one of three states: offline, online, or syncing.
Fault recovery process
The recovery sequence follows five steps:
- Detect node failure (within 3 seconds)
- Mark the node as offline
- Automatically elect a new leader
- Sync data to the replacement node
- Restore service
Recovery time objectives
| Scenario | RTO | RPO |
|---|---|---|
| Single node failure | < 30 seconds | 0 |
| Network partition | < 1 minute | < 1 minute |
Load balancing
Vgroup load balancing
Two SQL commands manage load distribution:
- Trigger rebalancing:
BALANCE VGROUP; - View cluster state:
SHOW CLUSTER;
Write load distribution
The load balancer distributes write requests using three strategies: round-robin, least-connections, and IP hash.
Network configuration
Network isolation
Use separate dedicated networks for cluster internal communication, client access, and storage (distributed storage).
Network quality requirements
| Metric | Requirement |
|---|---|
| Bandwidth | 10-gigabit network |
| Latency | < 1ms |
| Packet loss rate | < 0.01% |
Capacity planning
Replica count selection
| Business grade | Replicas | Rationale |
|---|---|---|
| Core business | 3 | High availability for production workloads |
| General business | 2 | Balance between cost and availability |
| Development/testing | 1 | Development and test environments only |
Node planning
Production deployments need at least three nodes. This supports three-replica deployment, ensures majority quorum elections, and eliminates single points of failure.
Summary
Five practices form the foundation of a production-ready TDengine high availability deployment:
- Choose replica strategies that match the workload
- Verify network quality before production rollout
- Set up monitoring and alerting
- Run regular failover drills
- Maintain disaster recovery and restoration plans
