log volume by host

[bgill0123]

Hello,

I have 26 hosts reporting data to a specific index. These hosts are prone to malfunction at any time 🙂

Is there a search I can do that will show me any dramatic increases in logging volume from any of the hosts?

Thanks

[manjunathmeti]

hi @bgill0123 ,

The below query will give the total count of events per host every hour for the index. You can select any chart to see the pattern.

| tstats count where index="indexname" by host, _time 
| timechart span=1h sum(count) as count by host

If this reply helps you, an upvote/like would be appreciated.

[effem2]

Hi bgill0123,

in order to get an overview of the data ingestion rates by host you can use this example:

index=_internal sourcetype=splunkd fwdType=* group=tcpin_connections(connectionType=cooked OR connectionType=cookedSSL) 
| timechart minspan=30s avg(eval(tcp_KBps)) as "KB/s", avg(tcp_eps) as "Events/s" by hostname

Additionally if you are using the Monitoring Console you can find desired information here:
https://<your_MC>:8000/en-GB/app/splunk_monitoring_console/forwarder_deployment

In order to only get the the dramatically increased ingestion-rates you need to define what "dramatic" means.
If we are going with "double" == "dramatic" then we can run the following example:

index=_internal sourcetype=splunkd "group=tcpin_connections" ("connectionType=cooked" OR "connectionType=cookedSSL")  earliest="-2d@d" latest=@d 
| fields hostname, tcp_KBps
| bin _time span=1h
| stats max(eval(tcp_KBps)) as "max_kbs" by _time, hostname 
| stats avg(max_kbs) as avg_max_kbs latest(max_kbs) as latest_max_kbs by hostname
| eval dramatic_threshold=avg_max_kbs*2
| eval dramatic=if(latest_max_kbs>dramatic_threshold,"true","false")
| where dramatic=="true"

This will give you a list of hosts where in the last 1h there was a peak higher than the average peaks in the last 2 days.

This can be modified to calculate the Volume as well. 

-----
Hope this helps.
If this answer helped you, please upvote/mark as resolution.

Kind,
Florian