topic Re: How do I tell if a forwarder is down? in Getting Data In

How do I tell if a forwarder is down?

Alan_Bradley — Sat, 20 Mar 2010 06:46:46 GMT

How can you differentiate between a forwarder being down and a forwarder not having any data to send ? i.e is there a heartbeat that i can tap into?

Re: How do I tell if a forwarder is down?

matt — Sat, 20 Mar 2010 06:49:59 GMT

If you have deployed a number of Splunk forwarders and they are all pushing data to Splunk, you might not notice if one of them goes out of service, because the other forwarders are still pushing data to Splunk. You can run the following search to detect forwarders that have been up in the last 24 hours but not in the last 2 minutes. It uses the forwarder heartbeat, which is a feature of Splunk versions 3.2 and later.

index=_internal sourcetype="fwd-hb" starthoursago=24 | dedup host | eval age = strftime("%s","now") - _time | search age > 120 age < 86000

You can set this search up as an alert every several minutes so that Splunk will let you know if any of your active forwarders have not responded in the last 2 minutes.

If you're running a version of Splunk that is later than 3.3, the heartbeat message is not longer sent. Use the following search instead:

The following search works in 3.4.5 and finds all hosts who haven't sent a message in the last 24 hours

and in 4.0:

Another 4.0 variant

| metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time

Caveat: Many of these methods do not account for decommissioned hosts, which you are bound to have after a length of time. These hosts will also show up in the search results, as they also fit the criteria. Incorporating a host tag ('decommisioned', etc) into this search may help with this, but requires you to tag known hosts that are no longer valid.

Re: How do I tell if a forwarder is down?

marcoscala — Thu, 15 Jul 2010 17:19:13 GMT

I'm going to test this later on 4.1.3 and let you know. I need to provide our Customer a dashboard to monitor all the remote forwarder at a glance.

Marco

Re: How do I tell if a forwarder is down?

marcoscala — Thu, 15 Jul 2010 22:00:21 GMT

Matt, I'v tried with Splunk 4.1.3 the following search:

and I got the following error:

"Error in 'eval' command: Typechecking failed. '-' only takes numbers."

Marco

Re: How do I tell if a forwarder is down?

marcoscala — Wed, 25 Aug 2010 20:35:32 GMT

Hi All!

I just made another test and changed a bit the logic.

I was looking for Forwarders not being sending data for more than let's say 2 minutes. Here's my latest version:

If you prefere, you can work on seconds of course instead of minutes 🙂

Marco

Re: How do I tell if a forwarder is down?

joseluisrespeto — Thu, 18 Dec 2014 11:46:09 GMT

Good morning,

How can i check that the forwarders are sending the logs correctly? I have the following error in my logs:

"eventType=connect_fail" in metrics.log

metrics.log:12-17-2014 09:38:48.529 +0100 INFO StatusMgr - destHost=10.26.XX.XX, destIp=10.26.XX.XX, destPort=9997, eventType=connect_fail, publisher=tcpout, sourcePort=8089, statusee=TcpOutputProcessor

This event produce that the los are not been sending correctly, them i need know if any option in the program execution can check if splunk is sending or not the data to the server.

And, can i resolve this issue in the configuration with some parameter? This issue only appear in determinate times isn't fixed.

Thanks and regards.

Re: How do I tell if a forwarder is down?

rameshyedurla — Tue, 29 Sep 2020 08:30:36 GMT

try this
| metadata type=hosts

| eval lastHour=relative_time(now(),"-1h@h")
| eval yesterday=relative_time(now(), "-1d@d")
| where ( recentTime>yesterday AND recentTime