A reliable log system is fundamental to operating any database in production. Logs capture what happened, when it happened, and how long it took. For a distributed time-series database like TDengine, the log system is also the first line of investigation when diagnosing performance issues, query latency, or cluster anomalies.
TDengine’s logging system has two main categories: general logs for runtime events and Slow Logs for performance diagnostics. This guide covers the architecture, configuration, and operational practices for both.
1. General log architecture
1.1 Asynchronous write mechanism
TDengine writes general logs asynchronously to avoid blocking the main execution path. When a log entry is generated, it is placed into an in-memory buffer and acknowledged immediately. A background thread handles the actual disk write. This design keeps the log system from becoming a performance bottleneck on busy clusters.
1.2 Circular buffer and dynamic refresh
The in-memory log buffer operates as a circular buffer with a 20 MB fixed size. When the buffer fills up, older entries are overwritten by newer ones before they are flushed to disk. This is safe because the flush thread continuously drains the buffer to persistent storage.
The flush interval is dynamic, adjusting between 5 ms and 25 ms based on buffer utilization:
- At low utilization: flush every 25 ms to reduce CPU overhead.
- At high utilization: flush every 5 ms to prevent buffer overflow and data loss.
This dynamic refresh mechanism balances throughput with resource consumption. Under normal load, the system runs at the slower interval. During log-heavy periods (such as cluster startup or failover events), the interval tightens automatically.
2. Slow Log system
2.1 What Slow Log captures
The Slow Log records database operations that exceed a configurable time threshold. By default, the threshold is 10 seconds, and the scope covers SELECT queries. Slow Log entries include:
- The full SQL statement
- Execution duration
- Affected database and table names
- Query plan information (if available)
- Client endpoint and user
Administrators can adjust what gets logged via the slowLogScope parameter:
ALL: Log all slow operations (queries, inserts, and others)QUERY: Log slow queries only (default)INSERT: Log slow inserts onlyOTHERS: Log slow operations other than queries and insertsNONE: Disable Slow Log entirely
2.2 Batch reporting with queue and timer
Slow Log entries use a batch reporting mechanism to reduce I/O overhead. Instead of writing each slow operation to disk individually, entries accumulate in a queue and are flushed in batches. The batch trigger follows a dual condition:
- Queue threshold: When 100 slow log entries accumulate, flush immediately.
- Timer threshold: If fewer than 100 entries are queued but 5 seconds have elapsed since the last flush, flush the current batch.
This design avoids both write amplification (from per-entry flushes) and stale log data (from waiting too long for a full queue).
2.3 Persistent storage via temp files
Before the final Slow Log file is written, entries are cached in temp files on disk. This provides a durability guarantee: if the database process crashes before the next flush cycle, Slow Log entries that were already written to temp files survive and are recovered on restart.
The temp file approach also helps during high-load scenarios. When the system is under memory pressure, temp files act as a spillover buffer, preventing Slow Log data from consuming application memory.
3. Log level configuration
3.1 Bit mask encoding
TDengine uses a bit mask to encode log levels. Each bit in the debugFlag parameter represents an output destination or a severity filter:
| Bit | Value | Meaning |
|---|---|---|
| 0 | 1 | Output to console (stdout) |
| 1 | 2 | Output to file |
| 2 | 4 | DEBUG level |
| 3 | 8 | TRACE level |
| 4 | 16 | INFO level |
| 5 | 32 | WARNING level |
| 6 | 64 | ERROR level |
| 7 | 128 | FATAL level |
To configure the log level, add the bit values for the desired combination. Common values:
debugFlag | Composition | Meaning |
|---|---|---|
| 131 | FILE + WARNING + ERROR (2 + 32 + 64 + …) | Production: file output, warnings and errors only |
| 135 | FILE + INFO + WARNING + ERROR | Production with info: standard operations visible |
| 143 | FILE + DEBUG + INFO + WARNING + ERROR | Troubleshooting: debug-level detail |
| 7 | CONSOLE + FILE + DEBUG (1 + 2 + 4) | Development: console and file with debug output |
| 3 | CONSOLE + FILE (1 + 2) | Minimal: console and file, OS-level severity only |
3.2 Configuration in taos.cfg
Set the log level in the taos.cfg configuration file:
# Production: file output, INFO and above
debugFlag 135
# Troubleshooting: file output, DEBUG and above
debugFlag 143
# Minimal: file output, WARNING and above
debugFlag 131
After changing debugFlag, restart the taosd service for the new value to take effect. The change applies to all dnodes in the cluster that read the same configuration file.
4. Log lifecycle management
4.1 Rotation and retention
TDengine rotates log files based on line count, not time. The numOfLogLines parameter sets the maximum number of lines per log file (default: 10,000,000). When a log file reaches this limit, it is closed and a new file is created.
Retention is controlled by logKeepDays (default: 0, meaning unlimited). When set to a positive integer, log files older than the specified number of days are deleted by the system. For production clusters, setting logKeepDays to 30 or 90 is common practice.
4.2 Disk space safeguards
TDengine enforces a minimum free space requirement for the log directory via minimalLogDirGB. If free space in the log directory falls below this threshold (default: 1 GB), the system stops writing logs and emits a warning. This prevents a full disk from causing database unavailability.
The log directory is configured separately from the data directory:
logDir /var/log/taos
dataDir /var/lib/taos
Keeping logs and data on separate volumes is recommended. A log volume filling up should not affect database write operations.
5. Log analysis with command-line tools
5.1 Finding slow queries
Extract Slow Log entries and sort by duration to identify the longest-running queries:
grep "slow log" /var/log/taos/taosdlog.* | awk '{print $NF, $0}' | sort -rn | head -20
5.2 Counting log entries by severity
Count WARNING and ERROR entries across all log files:
grep -c "WARNING\|ERROR" /var/log/taos/taosdlog.*
5.3 Tracking a specific operation
Filter log entries for a particular database or table name:
grep "db_name" /var/log/taos/taosdlog.*
5.4 Time-range analysis
Extract log entries within a specific time window. This is useful when correlating log events with a reported incident:
awk '/2026-05-28 10:00:00/,/2026-05-28 11:00:00/' /var/log/taos/taosdlog.0
5.5 Monitoring log output rate
Watch the log output rate in real time to detect abnormal activity:
tail -f /var/log/taos/taosdlog.0 | pv -l > /dev/null
6. Operations monitoring integration
6.1 Built-in monitoring via taosKeeper
TDengine includes taosKeeper, a monitoring component that exports cluster metrics to a dedicated TDengine database. Metrics include:
- dnode CPU, memory, and disk usage
- vnode write throughput and query latency
- Connection counts and request rates
- Stream Processing task status
- Data Subscription consumer lag
These metrics are stored as time-series data, meaning operators can query monitoring data with the same SQL tools they use for application data:
SELECT AVG(cpu_usage), MAX(cpu_usage)
FROM sys_monitor_dnode
WHERE fqdn = 'node1.tdengine.com'
INTERVAL(5m);
6.2 TDinsight Grafana dashboard
TDinsight is a pre-built Grafana dashboard that visualizes the monitoring data collected by taosKeeper. It provides panels for cluster health, node resource usage, query performance, write throughput, and Storage-Compaction status. TDinsight is the recommended starting point for production monitoring.
6.3 Alerting from log patterns
For proactive alerting, operators can combine log monitoring with external tools. A common pattern is to tail the log file and trigger alerts on specific patterns:
tail -f /var/log/taos/taosdlog.0 | grep --line-buffered "ERROR\|FATAL" | while read line; do
<em># Send alert via webhook, email, or messaging system</em>
curl -X POST -H "Content-Type: application/json" \
-d "{\"text\": \"TDengine alert: $line\"}" \
https://alerting-webhook.example.com/alert
done
