Create a high-performance fintech data center that stores massive amounts of market-related time-series data, including stocks, investment funds, bonds, and futures.
TDengine Solution
Background
Market data often exhibits the following characteristics:
- Scale: terabytes of data with complex storage requirements
- Characteristics: fixed format, timestamped
- Subtables: generally hundreds of thousands per deployment, with over 10 million possible in trade signal and factor analysis scenarios
- Retention period: generally five to ten years, though 30 years and longer are seen in some cases
Considering the characteristics of financial data – large scale, fixed structure, long retention period, and many tables – TDengine is an excellent choice for processing and storing this data. The recommended data model is to create one supertable for each time of financial instrument and one subtable for each specific metric. This is an effective way to manage huge amounts of market data and takes advantage of the high performance compute engine in the TDengine time-series database (TSDB) to provide investment and research services including asset management, real-time monitoring, performance analysis, risk analysis, portfolio backtesting, trade signal simulation (contracts, policies, etc.), and report generation.
Requirements and Pain Points
- Financial data needs more than high performance – it also demands high precision writes.
- There are many usage scenarios for financial data, and it is essential that querying data from any time period be fast.
- The monitoring of asset and derivative data also requires high standards for database performance.
- Quantitative applications such as backtesting and deep learning are performed over a large set of data, including simulated and preprocessed data.
Solution Architecture
With the variety of data ingestion and import methods that TDengine provides, you can write market data files and real-time data flows into TDengine and create your financial services through the SDK or over HTTP.
Benefits
The benefits of the solution are described as follows:
- Reduce data storage costs to one tenth of competing solutions through TDengine’s ultra-high compression ratios, using lossy compression for floating point numbers as well as two-pass lossless compression.
- Write 100 million data points per second with TDengine’s high-performance compute and storage engines and clustered architecture
- Eliminate downtime and data loss with TDengine’s high availability and strict data consistency, including validation for data replication, simultaneous data writes across nodes, and disk writes.
- Read data for a single asset in under 1 ms.
- Perform model training and verification on data from any time range without performance loss for historical data
Fintech Case Studies
