Splunk Search

Splunk search command for daily indexed volume?

remy06
Contributor

Hi,

I am currently testing out two searches to report and alert on the daily indexed volume.

The first search is as follows.:
index=_internal sourcetype=splunkd LicenseManager-Audit todaysBytesIndexed licenseSize | eval todayMb=(todaysBytesIndexed/1024/1024) | eval percentUsed=round(100*todayMb/licenseSize,2)

The second search will have:
index=_internal source=*metrics.log splunk_server="*" group="per_host_thruput" | eval MB=kb/1024 | chart sum(MB) by series | sort sum(MB) | addcoltotals col=t | fillnull value="[ Total Indexed Volume ] last 24 hours" Host(s)

Both are scheduled to run at midnight daily and should report on the indexed volume for that day.

However,I've noticed the reports for today,these 2 searches show different indexed volume which are misleading.

Which command is more accurate?Any suggestion or advice?

1 Solution

sideview
SplunkTrust
SplunkTrust

The LicenseManager search will not count things like index=_internal and index=_audit data, because that volume doesnt count against your license. And the per_host search does.

However you can use the per_index_thruput numbers and then filter out the indexes that have leading underscores.

index=_internal source=*metrics.log splunk_server="*" group="per_index_thruput" | eval MB=kb/1024 | stats sum(MB) by series | rename series as index | search index!=_* | sort sum(MB) | addcoltotals | fillnull value="[ Total Indexed Volume ] last 24 hours" index

If I run this search against the data from yesterday and compare it to the licenseManager's search from today (necessary because the licenseManager runs just after midnight and its talking about yesterday), then the numbers seem very close to eachother but oddly they are not equal. Im not sure why.

View solution in original post

yannK
Splunk Employee
Splunk Employee

Since Splunk 4.2, a new log record the license usage on the license-master.

you can use new searches for the detail per : sourcetype/host/source per pool
see http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume

sideview
SplunkTrust
SplunkTrust

The LicenseManager search will not count things like index=_internal and index=_audit data, because that volume doesnt count against your license. And the per_host search does.

However you can use the per_index_thruput numbers and then filter out the indexes that have leading underscores.

index=_internal source=*metrics.log splunk_server="*" group="per_index_thruput" | eval MB=kb/1024 | stats sum(MB) by series | rename series as index | search index!=_* | sort sum(MB) | addcoltotals | fillnull value="[ Total Indexed Volume ] last 24 hours" index

If I run this search against the data from yesterday and compare it to the licenseManager's search from today (necessary because the licenseManager runs just after midnight and its talking about yesterday), then the numbers seem very close to eachother but oddly they are not equal. Im not sure why.

jlaigo2
Path Finder

This is what I have been looking for thanks for your answer:)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...