Reporting

daily indexed volume search does not report all hosts

apro
Path Finder

Hi,

I've noticed that using the search command below to generate daily indexed volume, it doesn't display all the hosts that is still sending data to our splunk indexer.

index=_internal source=*metrics.log splunk_server="*" group="per_host_thruput"
| eval MB=kb/1024
| chart sum(MB) by series
| rename series AS "Host(s)"
| sort sum(MB)
| addcoltotals col=t
| fillnull value="[ Total Indexed Volume ] last 24 hours" Host(s)  

For example it listed only 10 hosts out of 15 hosts which I had.

But when I do a normal search on the missing hosts, I am able to see their current latest events.

0 Karma

apro
Path Finder

In this case,can you please advise on a search command that will generate a list of all hosts and their indexed volume on a daily basis?

I am trying to create a report to monitor the indexed volume like this:
hosts sum(MB)
----- ------
A 10.5
B 9.5 ..
..
Total 20

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Also, please edit your existing question instead of posting a new answer.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

this is not recorded, but you can approximate it with * | bucket _time span=1d | stats sum(len(_raw)) by _time,host, provided the indexed data was indexed in real time. You can get something more accurate if the data was indexed in real time with * | bucket _indextime span=1d as indextime | stats sum(len(_raw)) by indextime,host but this will require you to run the search over the entire time range of possibly indexed data.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Splunk metrics logging will never list every host (or every host or source or sourcetype), but only the top 10 in each 30-second interval. Therefore, the results reported by metrics logging are an approximation. The number of hosts can be increased by changing [metrics] maxseries in limits.conf, but that comes at the cost of larger internal logs.

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...