topic Re: How can I set an alert for max thruput? in Getting Data In

How can I set an alert for max thruput?

mamir32825 — Sun, 22 Oct 2017 14:03:33 GMT

What is the search query to alert when the forwarder reaches max thruput?

Re: How can I set an alert for max thruput?

inventsekar — Mon, 23 Oct 2017 07:45:12 GMT

from an answer by @burwell at the post - Here's what I do to detect throttled forwarders. I have a scheduled search for last 4 hours (-240m to now) and then alert for any events:

 index=_internal " INFO " " throttled" NOT debug source=*splunkd.log*  | dedup host |sort host|  table host _raw

This gives me a nice table per host and I can see the hosts and what the thruput is that is getting throttled. Example output:

 foo1.host.com        10-22-2017 18:26:28.131 +0000 INFO ThruputProcessor - Current data throughput (258 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
 foo2.host.com        10-22-2017 18:29:28.324 +0000 INFO ThruputProcessor - Current data throughput (512 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.

Re: How can I set an alert for max thruput?

gcusello — Mon, 23 Oct 2017 07:53:59 GMT

Hi
In Distributed Monitoring Console you can find a dashboard panel that list forwarder's connections.

| inputlookup dmc_forwarder_assets 
| eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps) 
| fields hostname status avg_tcp_kbps
| where avg_tcp_kbps>threeshold

From this search you can find the Average kb/s and put a threeshold alert.

Bye.
Giuseppe

Re: How can I set an alert for max thruput?

gjanders — Tue, 24 Oct 2017 02:26:52 GMT

I have an example in this savedsearches.conf from the SplunkAdmins app that I created. My search is similar to other answers but I have used:

index=_internal "has reached maxKBps. As a result, data forwarding may be throttled" sourcetype=splunkd
| stats count(_raw) by host as countPerHost 
| where countPerHost > 1