Splunk Search

log volume by host

bgill0123
New Member

Hello,

I have 26 hosts reporting data to a specific index. These hosts are prone to malfunction at any time 🙂

Is there a search I can do that will show me any dramatic increases in logging volume from any of the hosts?

 

Thanks

Labels (1)
0 Karma

manjunathmeti
Champion

hi @bgill0123 ,

The below query will give the total count of events per host every hour for the index. You can select any chart to see the pattern.

| tstats count where index="indexname" by host, _time 
| timechart span=1h sum(count) as count by host

 

If this reply helps you, an upvote/like would be appreciated.

0 Karma

effem2
Explorer

Hi bgill0123,

 

in order to get an overview of the data ingestion rates by host you can use this example:

 

index=_internal sourcetype=splunkd fwdType=* group=tcpin_connections(connectionType=cooked OR connectionType=cookedSSL) 
| timechart minspan=30s avg(eval(tcp_KBps)) as "KB/s", avg(tcp_eps) as "Events/s" by hostname

 

 

Additionally if you are using the Monitoring Console you can find desired information here:
https://<your_MC>:8000/en-GB/app/splunk_monitoring_console/forwarder_deployment

In order to only get the the dramatically increased ingestion-rates you need to define what "dramatic" means.
If we are going with "double" == "dramatic" then we can run the following example:

 

index=_internal sourcetype=splunkd "group=tcpin_connections" ("connectionType=cooked" OR "connectionType=cookedSSL")  earliest="-2d@d" latest=@d 
| fields hostname, tcp_KBps
| bin _time span=1h
| stats max(eval(tcp_KBps)) as "max_kbs" by _time, hostname 
| stats avg(max_kbs) as avg_max_kbs latest(max_kbs) as latest_max_kbs by hostname
| eval dramatic_threshold=avg_max_kbs*2
| eval dramatic=if(latest_max_kbs>dramatic_threshold,"true","false")
| where dramatic=="true"


This will give you a list of hosts where in the last 1h there was a peak higher than the average peaks in the last 2 days.

This can be modified to calculate the Volume as well. 

-----
Hope this helps.
If this answer helped you, please upvote/mark as resolution.

Kind,
Florian

 

Tags (3)
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!