Data Retention

Performance Data

Time-series, or performance, data is data made up of a series of values collected for a particular metric over time. This data can then be plotted on a graph for a specified time period to show relative performance over that time period. Netreo stores this time-series data in individual database files for each metric (This includes performance data that it collects directly from managed devices and that it calculates itself, such as for log or WebART check performance).

The first time a managed device is polled, Netreo's data polling engine creates a new database file for each statistic being collected. The structure of this file is such that it does not grow in storage size as new data is added. Instead, it "rolls up" older data into averages of itself, freeing up space for the newer raw data values. Data that has reached its retention time limit (see tables below) is no longer rolled-up and is simply overwritten with the most recent value. This ensures that Netreo's data footprint does not grow unless you add more managed devices (thus more database files). As a result, Netreo can retain data for any given metric for several years without changing the file size of the database file. However, as you look farther back in time, the data becomes less granular and more averaged.

By default, Netreo retains data for the following periods:

General Data Retention

Per-process Data Retention
(CPU per process, memory usage per process)

Note: The retention periods specified in the tables above apply only to resources that are being actively polled for data. For resources that are disabled or otherwise inactive, data is retained for no longer than 3 months before it is deleted to conserve space.

When displaying time-series data in graphs, Netreo always shows the most granular data (shortest collection period) available that covers the entire time period of the graph. Therefore a report run in Netreo with a start date of 120 days ago will show data from the 30-minute average (the most granular for that time period). Zooming into that graph will show the most granular data available for the zoomed time period, allowing you to quickly get to the most granular data level available.

Keep in mind that even a graph zoomed-in to maximum will only display data as granular as is available for that historical period. To zoom in to 1-minute data, a graph must only be displaying data that is less than 7 days old. If any data points older than that are included on the graph, it will revert to 5-minute data. To zoom in to 5-minute data, a graph must only be displaying data that is less than 100 days old. To zoom in to 30-minute data, a graph must only display data that is less than 196 days old. And so on.

Note regarding service engines: Performance data collected by a remote poller service engine is also retained on a rolling basis on that service engine for 45 minutes. In the event that the Netreo primary appliance loses its connection to the service engine, as long as the disruption lasts for less than 45 minutes, no performance data is lost.

Averaging Calculations

This section explains how missing and late data values affect Netreo’s averaging calculations.

The method used by Netreo to store and display averaged data values can be affected by NaNs ("Not a Number") introduced into the collected data sets. NaNs are typically caused by missing or late data values. When a NaN is introduced into collected data, it will cause a small discrepancy between the stored/displayed value of an average as calculated by Netreo and the actual average as calculated using the original raw data produced by the data source.

The image below helps to explain how Netreo's averaging calculations are affected in different situations.

Note: This diagram assumes a data collection interval of 15 minutes, with averaging of values taking place every at intervals of 60 minutes and 3 hours. It is important to understand that this diagram is not a direct representation of Netreo's actual data collection and averaging intervals. The intervals involved and the number of data values shown in the image are for illustrative purposes only. In reality, Netreo collects many more data values than shown before averaging them over intervals different from those in the diagram. However, the sheer number of values actually involved would make this diagram difficult to read and understand, so it is highly simplified to make the concept more clear. The actual number of data values involved and the intervals over which they are averaged by Netreo are outlined at the top of this page.

Normal Data Collection

The first section, highlighted in green, shows an ideal data collection cycle. Each data value (D1, D2, D3, D4) is collected on time (every 15 minutes in this example) and recorded in the correct 15-minute bucket. After 60 minutes, the values from all four 15-minute buckets are averaged, and that value is recorded in the 60-minute bucket. Under ideal circumstances, the same would happen for the next three 60-minute buckets. After 3 hours, the values from all three 60-minute buckets are averaged, and that value is recorded in the 3-hour bucket.

Late Data

The second section, highlighted in blue, shows what happens when a data value (D3 in this example) is not received during its expected 15-minute interval but is received at the next 15-minute interval along with the data value collected normally during that interval (D4). The 15-minute bucket where D3 should have been stored now contains a NaN value (because D3 was not received in time), while the next 15-minute bucket now contains the average of data values D3 and D4, which were both received during that interval. (The storage buckets will always average however many data values were received during their particular time interval. In this example it was two data values, D3 and D4, but it could have been any number of values.) After 60 minutes, the values from all four 15-minute buckets are averaged as the average of D1, D2, and the average of D3 and D4 (the NaN is disregarded), and that value is recorded in the 60-minute bucket. The key concept to understand here is that the average of D1, D2, and the average of D3 and D4 is not equal to the average of D1, D2, D3, and D4. There would be a small discrepancy between the actual average of those data values and the value now recorded in the 60-minute bucket. After 3 hours, when the values from the 60-minute buckets are averaged, the value recorded in the 3-hour bucket will be very minutely off.

Missing Data

The third section, highlighted in red, shows what happens when a data value (D2 in this example) is never received at all. (This example assumes D1, D3, and D4 are received normally.) This time, the 15-minute bucket where D2 should have been stored now contains a NaN value (since D2 was never received). After 60 minutes, the values from all four 15-minute buckets are averaged as the average of D1, D3, and D4 (the NaN is disregarded), and that value is recorded in the 60-minute bucket. There would be a small discrepancy between the average of the data values produced by the data source and the average of those received data values now recorded in the 60-minute bucket. After 3 hours, when the values from the 60-minute buckets are averaged, the value recorded in the 3-hour bucket will be very minutely off.

Netreo's Actual Data Collection and Averaging Intervals

The data collection and averaging intervals used in the image above are simplified for convenience and ease of display. Once the data averaging (roll-up) concept is understood, it can be applied to Netreo's actual data collection and averaging intervals, which are specified in the tables on this page.

Log-Style Data

For historical-log-type data (such as Syslog, alert history, audit logs, detailed call records, and the like), Netreo stores the last 3 million records, regardless of date. Each log type is handled independently, so Netreo stores 3 million records for each type (except SNMP trap and Syslog—which are stored together). In most environments, this provides several years of log history. However, in very high-traffic environments, it may be less. Netreo stores time-series data for log performance monitoring as indicated in the time-series data section above.

Traffic Flow Data

Traffic flow data is accounting type data on specific IP connections. This type of data is provided by technologies such as NetFlow, sFlow, and IPFIX.

Netreo stores this data in two ways:

The detailed 1-minute flow logs, which are stored for 6 hours, include specific information on individual TCP connections between devices. Also, time-series flow data (overall volume per application per interface) is stored as indicated in the time-series data section above.

Please contact Netreo support to discuss your requirements if you need additional flow log storage. Some environments may require additional hard disk capacity to accommodate larger flow archives.