Splunk Enterprise

Data Age Search

mwdbhyat
Builder

Hi there,

Ive expanded the data age vs frozen age dash into search and want to get an overview of the whole platform listing each index with its oldest data etc.

| rest splunk_server_group=dmc_group_indexer splunk_server_group="*" /services/data/indexes/
| join title splunk_server index type=outer [| rest splunk_server_group=dmc_group_indexer splunk_server_group="*" /services/data/indexes-extended/]
| eval bucketCount = coalesce(total_bucket_count, 0)
| eval eventCount = coalesce(totalEventCount, 0)
| eval coldBucketSize = coalesce('bucket_dirs.cold.bucket_size', 'bucket_dirs.cold.size', 0)
| eval coldBucketSizeGB = round(coldBucketSize/ 1024, 2)
| eval coldBucketMaxSizeGB = if(isnull('coldPath.maxDataSizeMB') OR 'coldPath.maxDataSizeMB' = 0, "unlimited", round('coldPath.maxDataSizeMB' / 1024, 2))
| eval coldBucketUsageGB = coldBucketSizeGB." / ".coldBucketMaxSizeGB
| eval homeBucketSizeGB = coalesce(round((total_size - coldBucketSize) / 1024, 2), 0.00)
| eval homeBucketMaxSizeGB = round('homePath.maxDataSizeMB' / 1024, 2)
| eval homeBucketMaxSizeGB = if(homeBucketMaxSizeGB > 0, homeBucketMaxSizeGB, "unlimited")
| eval homeBucketUsageGB = homeBucketSizeGB." / ".homeBucketMaxSizeGB
| eval dataAgeDays = coalesce(round((now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")) / 86400, 0), 0)
| eval frozenTimePeriodDays = round(frozenTimePeriodInSecs / 86400, 0)
| eval frozenTimePeriodDays = if(frozenTimePeriodDays > 0, frozenTimePeriodDays, "unlimited")
| eval freezeRatioDays = dataAgeDays." / ".frozenTimePeriodDays
| eval indexSizeGB = if(currentDBSizeMB >= 1 AND totalEventCount >=1, round(currentDBSizeMB/1024, 2), 0.00)
| eval maxTotalDataSizeGB = round(maxTotalDataSizeMB / 1024, 2)
| eval indexMaxSizeGB = if(maxTotalDataSizeGB > 0, maxTotalDataSizeGB, "unlimited")
| eval indexSizeUsageGB = indexSizeGB." / ".indexMaxSizeGB
| eval indexSizeUsagePerc = if(isNum(indexMaxSizeGB) AND (indexMaxSizeGB > 0), round(indexSizeGB / indexMaxSizeGB * 100, 2)."%", "N/A")
| eval total_raw_size = coalesce(total_raw_size, 0) | eval avgBucketSize = round(indexSizeGB / bucketCount, 2)
| fields title, splunk_server, freezeRatioDays, indexSizeUsageGB, homeBucketUsageGB, coldBucketUsageGB, eventCount, bucketCount, avgBucketSize
| rename splunk_server as "Indexer"
freezeRatioDays as "Data Age vs Frozen Age (days)"
indexSizeUsageGB as "Index Usage (GB)"
homeBucketUsageGB as "Home Path Usage (GB)"
coldBucketUsageGB as "Cold Path Usage (GB)"
eventCount as "Total Event Count"
bucketCount as "Total Bucket Count"
avgBucketSize as "Average Bucket Size (GB)"

I have the index name etc listed, but how do i group all the indexers so that the indexes are not split by each indexer, but rather just one "overall" view by index?

Thanks!

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Group data using the stats command.

... | stats values(*) as * by Indexer

 

---
If this reply helps you, Karma would be appreciated.
0 Karma

mwdbhyat
Builder

Hi Rich,

Thanks for the response, although thats not entirely what im after. I want to know all the info in that search across the whole cluster.. So as long as it tells me the data age for an index(as well as the other indexes.conf settings in that search), on ANY indexer somewhere. I understand that some sizes and usage will be different per indexer because of data distribution and such but i am just trying to get an overview of the cluster as a whole in terms of data retention by index - oldest event per index + overall index usage per index as a clusterwide whole.  

 

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...