Data Age Search

[mwdbhyat]

Hi there,

Ive expanded the data age vs frozen age dash into search and want to get an overview of the whole platform listing each index with its oldest data etc.

| rest splunk_server_group=dmc_group_indexer splunk_server_group="*" /services/data/indexes/
| join title splunk_server index type=outer [| rest splunk_server_group=dmc_group_indexer splunk_server_group="*" /services/data/indexes-extended/]
| eval bucketCount = coalesce(total_bucket_count, 0)
| eval eventCount = coalesce(totalEventCount, 0)
| eval coldBucketSize = coalesce('bucket_dirs.cold.bucket_size', 'bucket_dirs.cold.size', 0)
| eval coldBucketSizeGB = round(coldBucketSize/ 1024, 2)
| eval coldBucketMaxSizeGB = if(isnull('coldPath.maxDataSizeMB') OR 'coldPath.maxDataSizeMB' = 0, "unlimited", round('coldPath.maxDataSizeMB' / 1024, 2))
| eval coldBucketUsageGB = coldBucketSizeGB." / ".coldBucketMaxSizeGB
| eval homeBucketSizeGB = coalesce(round((total_size - coldBucketSize) / 1024, 2), 0.00)
| eval homeBucketMaxSizeGB = round('homePath.maxDataSizeMB' / 1024, 2)
| eval homeBucketMaxSizeGB = if(homeBucketMaxSizeGB > 0, homeBucketMaxSizeGB, "unlimited")
| eval homeBucketUsageGB = homeBucketSizeGB." / ".homeBucketMaxSizeGB
| eval dataAgeDays = coalesce(round((now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")) / 86400, 0), 0)
| eval frozenTimePeriodDays = round(frozenTimePeriodInSecs / 86400, 0)
| eval frozenTimePeriodDays = if(frozenTimePeriodDays > 0, frozenTimePeriodDays, "unlimited")
| eval freezeRatioDays = dataAgeDays." / ".frozenTimePeriodDays
| eval indexSizeGB = if(currentDBSizeMB >= 1 AND totalEventCount >=1, round(currentDBSizeMB/1024, 2), 0.00)
| eval maxTotalDataSizeGB = round(maxTotalDataSizeMB / 1024, 2)
| eval indexMaxSizeGB = if(maxTotalDataSizeGB > 0, maxTotalDataSizeGB, "unlimited")
| eval indexSizeUsageGB = indexSizeGB." / ".indexMaxSizeGB
| eval indexSizeUsagePerc = if(isNum(indexMaxSizeGB) AND (indexMaxSizeGB > 0), round(indexSizeGB / indexMaxSizeGB * 100, 2)."%", "N/A")
| eval total_raw_size = coalesce(total_raw_size, 0) | eval avgBucketSize = round(indexSizeGB / bucketCount, 2)
| fields title, splunk_server, freezeRatioDays, indexSizeUsageGB, homeBucketUsageGB, coldBucketUsageGB, eventCount, bucketCount, avgBucketSize
| rename splunk_server as "Indexer"
freezeRatioDays as "Data Age vs Frozen Age (days)"
indexSizeUsageGB as "Index Usage (GB)"
homeBucketUsageGB as "Home Path Usage (GB)"
coldBucketUsageGB as "Cold Path Usage (GB)"
eventCount as "Total Event Count"
bucketCount as "Total Bucket Count"
avgBucketSize as "Average Bucket Size (GB)"

I have the index name etc listed, but how do i group all the indexers so that the indexes are not split by each indexer, but rather just one "overall" view by index?

Thanks!

[richgalloway]

Group data using the stats command.

... | stats values(*) as * by Indexer

[mwdbhyat]

Hi Rich,

Thanks for the response, although thats not entirely what im after. I want to know all the info in that search across the whole cluster.. So as long as it tells me the data age for an index(as well as the other indexes.conf settings in that search), on ANY indexer somewhere. I understand that some sizes and usage will be different per indexer because of data distribution and such but i am just trying to get an overview of the cluster as a whole in terms of data retention by index - oldest event per index + overall index usage per index as a clusterwide whole.