Reporting

daily indexed volume search does not report all hosts

apro
Path Finder

Hi,

I've noticed that using the search command below to generate daily indexed volume, it doesn't display all the hosts that is still sending data to our splunk indexer.

index=_internal source=*metrics.log splunk_server="*" group="per_host_thruput"
| eval MB=kb/1024
| chart sum(MB) by series
| rename series AS "Host(s)"
| sort sum(MB)
| addcoltotals col=t
| fillnull value="[ Total Indexed Volume ] last 24 hours" Host(s)  

For example it listed only 10 hosts out of 15 hosts which I had.

But when I do a normal search on the missing hosts, I am able to see their current latest events.

0 Karma

apro
Path Finder

In this case,can you please advise on a search command that will generate a list of all hosts and their indexed volume on a daily basis?

I am trying to create a report to monitor the indexed volume like this:
hosts sum(MB)
----- ------
A 10.5
B 9.5 ..
..
Total 20

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Also, please edit your existing question instead of posting a new answer.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

this is not recorded, but you can approximate it with * | bucket _time span=1d | stats sum(len(_raw)) by _time,host, provided the indexed data was indexed in real time. You can get something more accurate if the data was indexed in real time with * | bucket _indextime span=1d as indextime | stats sum(len(_raw)) by indextime,host but this will require you to run the search over the entire time range of possibly indexed data.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Splunk metrics logging will never list every host (or every host or source or sourcetype), but only the top 10 in each 30-second interval. Therefore, the results reported by metrics logging are an approximation. The number of hosts can be increased by changing [metrics] maxseries in limits.conf, but that comes at the cost of larger internal logs.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Network to App: Observability Unlocked [May & June Series]

In today’s digital landscape, your environment is no longer confined to the data center. It spans complex ...

SPL2 Deep Dives, AppDynamics Integrations, SAML Made Simple and Much More on Splunk ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...