Getting Data In
Highlighted

How to create an alert when a new forwarder is added to deployment server?

Explorer

I'm looking for a way to report/alert anytime a new forwarder is added to my deployment server. I've tried searching on internal with the following search, but this isn't unique to when a forwarder is initially added to our environment.

index=_internal sourcetype=splunkd component=PubSubSvr host=<deploymentServer>
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

SplunkTrust
SplunkTrust

Hello merrelr,
you can create a lookup table of all forwarders:
| tstats count where index = _* by host | table host | outputlookup forwarders.csv
and let it run as a schedule search. now you can search and compare results to lookup to check who is new on your forwarders list:

   | tstats count where index = _* by host | search NOT [inputlookup forwarders.csv | fields+ host]
     | stats values(host) AS new_hosts

you can also use the | metadata command and search for newly received data leveraging the firstTime field
more in this answer:
https://answers.splunk.com/answers/321024/compare-search-to-lookup-table-and-return-results.html

0 Karma
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

New Member

This is very helpful!

Although when i enter the query to compare the results, I get the below error.

"Search Factory: Unknown search command 'not'."

I tried playing around with the query but did not have any luck.

0 Karma
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

New Member

I get the same error, "Search Factory: Unknown search command 'not'."

0 Karma
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

New Member

I found the answer. "search" is missing. Use
| tstats count where index = * by host | search NOT [inputlookup forwarders.csv | fields+ host]
| stats values(host) AS new
hosts

0 Karma
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

New Member

This worked! Thank you!

0 Karma
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

SplunkTrust
SplunkTrust

@bbraun
modified the answer to reflect, indeed forgot to put the search after pie. this is the reason for your error, there is no NOT command

0 Karma
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

New Member
| tstats count where index = _* by host | search NOT [| inputlookup forwarders.csv | fields+ host]
      | stats values(host) AS new_hosts

There should be a pipe before inputlookup.

0 Karma
Highlighted

Re: How to create an alert when a new forwarder is added to deployment server?

Motivator

Greetings @merrelr,

The current provided answers would work. This is how I would do it though:

| metadata type=hosts
| eval daysSinceFirstTime = round((now() - firstTime)/86400, 2)
| eval hoursSinceLastTime = round((now() - lastTime )/3600 , 2)
| convert ctime(firstTime) as firstTime
| convert ctime(lastTime)  as lastTime
| search daysSinceFirstTime < 1 OR (hoursSinceLastTime>24 AND hoursSinceLastTime<48)

In this example, you could run the alert search once a day to get all hosts that sent their first event to your indexer(s) in the past day and hosts that sent their last event to your indexer(s) between 24 and 48 hours ago. Run the search for all time as this is an extremely efficient command.

Refer here for the metadata command documentation.

Cheers,
Jacob
0 Karma