topic How to create an alert when a new forwarder is added to deployment server? in Getting Data In

How to create an alert when a new forwarder is added to deployment server?

merrelr — Mon, 01 May 2017 17:44:41 GMT

I'm looking for a way to report/alert anytime a new forwarder is added to my deployment server. I've tried searching on internal with the following search, but this isn't unique to when a forwarder is initially added to our environment.

index=_internal sourcetype=splunkd component=PubSubSvr host=<deploymentServer>

Re: How to create an alert when a new forwarder is added to deployment server?

adonio — Mon, 01 May 2017 17:58:41 GMT

Hello merrelr,
you can create a lookup table of all forwarders:
| tstats count where index = _* by host | table host | outputlookup forwarders.csv
and let it run as a schedule search. now you can search and compare results to lookup to check who is new on your forwarders list:

   | tstats count where index = _* by host | search NOT [inputlookup forwarders.csv | fields+ host]
     | stats values(host) AS new_hosts

you can also use the | metadata command and search for newly received data leveraging the firstTime field
more in this answer:
https://answers.splunk.com/answers/321024/compare-search-to-lookup-table-and-return-results.html

Re: How to create an alert when a new forwarder is added to deployment server?

bbraun — Wed, 17 May 2017 19:15:50 GMT

This is very helpful!

Although when i enter the query to compare the results, I get the below error.

"Search Factory: Unknown search command 'not'."

I tried playing around with the query but did not have any luck.

Re: How to create an alert when a new forwarder is added to deployment server?

jlh3dz — Tue, 20 Jun 2017 18:01:44 GMT

I get the same error, "Search Factory: Unknown search command 'not'."

Re: How to create an alert when a new forwarder is added to deployment server?

jlh3dz — Tue, 29 Sep 2020 14:32:17 GMT

I found the answer. "search" is missing. Use
| tstats count where index = _* by host | search NOT [inputlookup forwarders.csv | fields+ host]
| stats values(host) AS new_hosts

Re: How to create an alert when a new forwarder is added to deployment server?

bbraun — Wed, 21 Jun 2017 20:22:03 GMT

This worked! Thank you!

Re: How to create an alert when a new forwarder is added to deployment server?

adonio — Wed, 21 Jun 2017 20:57:21 GMT

@bbraun
modified the answer to reflect, indeed forgot to put the search after pie. this is the reason for your error, there is no NOT command

Re: How to create an alert when a new forwarder is added to deployment server?

buckiboy — Thu, 01 Aug 2019 13:11:57 GMT

| tstats count where index = _* by host | search NOT [| inputlookup forwarders.csv | fields+ host]
      | stats values(host) AS new_hosts

There should be a pipe before inputlookup.

Re: How to create an alert when a new forwarder is added to deployment server?

jacobpevans — Thu, 01 Aug 2019 14:01:38 GMT

Greetings @merrelr,

The current provided answers would work. This is how I would do it though:

| metadata type=hosts
| eval daysSinceFirstTime = round((now() - firstTime)/86400, 2)
| eval hoursSinceLastTime = round((now() - lastTime )/3600 , 2)
| convert ctime(firstTime) as firstTime
| convert ctime(lastTime)  as lastTime
| search daysSinceFirstTime < 1 OR (hoursSinceLastTime>24 AND hoursSinceLastTime<48)

In this example, you could run the alert search once a day to get all hosts that sent their first event to your indexer(s) in the past day and hosts that sent their last event to your indexer(s) between 24 and 48 hours ago. Run the search for all time as this is an extremely efficient command.

Refer here for the metadata command documentation.

Re: How to create an alert when a new forwarder is added to deployment server?

Tyler — Fri, 04 Mar 2022 20:00:09 GMT

Edit: This one should be more accurate and will also let you use the date picker.

index=_internal host="SPLUNK_DS_NAME" component=PubSubSvr event_message="Subscribed*" | eval hostname=mvindex(split(connectionId, "_"), 4) | search hostname!="direct" | eval guid=mvindex(split(connectionId, "_"), 5) | dedup guid sortby +_time | join guid type=inner [search index=_internal host="SPLUNK_DS_NAME" component=PubSubSvr event_message="Subscribed*" earliest=1 | eval hostname=mvindex(split(connectionId, "_"), 4) | search hostname!="direct" | eval guid=mvindex(split(connectionId, "_"), 5) | eval h_time=_time | dedup guid sortby +_time | fields guid, h_time] | addinfo | eval firstSeenDiff=_time-h_time | eval pickerDiff=now()-info_min_time | where pickerDiff>=firstSeenDiff | eval lastSeen=strftime(_time, "%Y-%m-%d %H:%M:%S") | eval firstSeen=strftime(h_time, "%Y-%m-%d %H:%M:%S") | table lastSeen, firstSeen, guid, hostname

This seems to be the earliest date stamp of events, not the forwarder check-in.

index=_internal source=*metrics.log group=tcpin_connections | dedup guid | table _time, guid, hostname

(Simplified version of Solved: Re: Listing forwarders - Splunk Community)

~~I then realized this was just for forwarders sending data. Although it would be strange, you can have forwarders connected to just the DS.~~

index=_internal host="SPLUNK_DS_HOSTNAME" component=PubSubSvr event_message="Subscribed*" | eval hostname=mvindex(split(connectionId, "_"), 4) | search hostname!="direct" | eval guid=mvindex(split(connectionId, "_"), 5) | dedup guid sortby +_time | table _time, guid, hostname