I've noticed that using the search command below to generate daily indexed volume, it doesn't display all the hosts that is still sending data to our splunk indexer.
index=_internal source=*metrics.log splunk_server="*" group="per_host_thruput" | eval MB=kb/1024 | chart sum(MB) by series | rename series AS "Host(s)" | sort sum(MB) | addcoltotals col=t | fillnull value="[ Total Indexed Volume ] last 24 hours" Host(s)
For example it listed only 10 hosts out of 15 hosts which I had.
But when I do a normal search on the missing hosts, I am able to see their current latest events.
In this case,can you please advise on a search command that will generate a list of all hosts and their indexed volume on a daily basis?
I am trying to create a report to monitor the indexed volume like this:
B 9.5 ..
this is not recorded, but you can approximate it with
* | bucket _time span=1d | stats sum(len(_raw)) by _time,host, provided the indexed data was indexed in real time. You can get something more accurate if the data was indexed in real time with
* | bucket _indextime span=1d as indextime | stats sum(len(_raw)) by indextime,host but this will require you to run the search over the entire time range of possibly indexed data.
Splunk metrics logging will never list every host (or every host or source or sourcetype), but only the top 10 in each 30-second interval. Therefore, the results reported by metrics logging are an approximation. The number of hosts can be increased by changing
[metrics] maxseries in limits.conf, but that comes at the cost of larger internal logs.