Hello,
I have 26 hosts reporting data to a specific index. These hosts are prone to malfunction at any time 🙂
Is there a search I can do that will show me any dramatic increases in logging volume from any of the hosts?
Thanks
hi @bgill0123 ,
The below query will give the total count of events per host every hour for the index. You can select any chart to see the pattern.
| tstats count where index="indexname" by host, _time
| timechart span=1h sum(count) as count by host
If this reply helps you, an upvote/like would be appreciated.
Hi bgill0123,
in order to get an overview of the data ingestion rates by host you can use this example:
index=_internal sourcetype=splunkd fwdType=* group=tcpin_connections(connectionType=cooked OR connectionType=cookedSSL)
| timechart minspan=30s avg(eval(tcp_KBps)) as "KB/s", avg(tcp_eps) as "Events/s" by hostname
Additionally if you are using the Monitoring Console you can find desired information here:
https://<your_MC>:8000/en-GB/app/splunk_monitoring_console/forwarder_deployment
In order to only get the the dramatically increased ingestion-rates you need to define what "dramatic" means.
If we are going with "double" == "dramatic" then we can run the following example:
index=_internal sourcetype=splunkd "group=tcpin_connections" ("connectionType=cooked" OR "connectionType=cookedSSL") earliest="-2d@d" latest=@d
| fields hostname, tcp_KBps
| bin _time span=1h
| stats max(eval(tcp_KBps)) as "max_kbs" by _time, hostname
| stats avg(max_kbs) as avg_max_kbs latest(max_kbs) as latest_max_kbs by hostname
| eval dramatic_threshold=avg_max_kbs*2
| eval dramatic=if(latest_max_kbs>dramatic_threshold,"true","false")
| where dramatic=="true"
This will give you a list of hosts where in the last 1h there was a peak higher than the average peaks in the last 2 days.
This can be modified to calculate the Volume as well.
-----
Hope this helps.
If this answer helped you, please upvote/mark as resolution.
Kind,
Florian