Progressive TTL
Progressive TTL provides a capability to manage data retention with different TTL settings and Segment Creation Policies based on the time granularity of metrics.
BanyanDB employs a Time-To-Live (TTL) mechanism to automatically delete data older than the specified duration. When using BanyanDB as the storage backend, the recordDataTTL
and metricsDataTTL
configurations are deprecated. Instead, TTL settings should be configured directly within storage.banyandb
.
For detailed information, please refer to the Storage BanyanDB documentation.
Segment Interval and TTL
BanyanDB’s data rotation mechanism manages data storage based on Segment Interval and TTL settings:
- Segment Interval (
SIDays
): Specifies the time interval in days for creating a new data segment. Segments are time-based, facilitating efficient data retention and querying. - TTL (
TTLDays
): Defines the time-to-live for data within a group, in days. Data that exceeds the TTL will be automatically deleted.
Best Practices for Setting SIDays
and TTLDays
- Data Retention Requirements: Set the TTL based on how long you need to retain your data. For instance, to retain data for 30 days, set the TTL to 30 days.
- Segment Management: Avoid generating too many segments, as this increases the overhead for data management and querying.
- Query Requirements: Align segment intervals with your query patterns. For example:
- If you frequently query data for the last 30 minutes, set
SIDays
to 1 day. - For querying data from the last 7 days, set
SIDays
to 7 days.
- If you frequently query data for the last 30 minutes, set
Configuration Guidelines
Record Data
For both standard and super datasets:
- Recommended
SIDays
:1
- Most queries are performed within a day.
TTLDays
: Set according to your data retention needs.
Metrics Data
Configure SIDays
and TTLDays
based on data retention and query requirements. Recommended settings include:
Group | SIDays |
TTLDays |
---|---|---|
Minute (gmMinute ) |
1 | 7 |
Hour (gmHour ) |
5 | 15 |
Day (gmDay ) |
15 | 15 |
Index (gmIndex ) |
15 | 15 |
Group Descriptions:
- Minute (
gmMinute
): Stores metrics with a 1-minute granularity. Suitable for recent data queries requiring minute-level detail. Consequently, it has shorterSIDays
andTTLDays
compared to other groups. - Hour (
gmHour
): Stores metrics with a 1-hour granularity. Designed for queries that need hour-level detail over a longer period than minute-level data. - Day (
gmDay
): Stores metrics with a 1-day granularity. This group handles the longest segment intervals and TTLs among all granularity groups. - Index (
gmIndex
): Stores metrics used solely for indexing without value columns. Since queries often scan all segments in theindex
group, it shares the sameSIDays
andTTLDays
as theday
group to optimize performance. This group’sTTL
must be set to the max value of all groups.
For more details on configuring segmentIntervalDays
and ttlDays
, refer to the BanyanDB Rotation documentation.