I'm looking for a way to report/alert anytime a new forwarder is added to my deployment server. I've tried searching on internal with the following search, but this isn't unique to when a forwarder is initially added to our environment.
index=_internal sourcetype=splunkd component=PubSubSvr host=<deploymentServer>
you can create a lookup table of all forwarders:
| tstats count where index = _* by host | table host | outputlookup forwarders.csv
and let it run as a schedule search. now you can search and compare results to lookup to check who is new on your forwarders list:
| tstats count where index = _* by host | search NOT [inputlookup forwarders.csv | fields+ host] | stats values(host) AS new_hosts
you can also use the
| metadata command and search for newly received data leveraging the firstTime field
more in this answer:
This is very helpful!
Although when i enter the query to compare the results, I get the below error.
"Search Factory: Unknown search command 'not'."
I tried playing around with the query but did not have any luck.
I found the answer. "search" is missing. Use
| tstats count where index = * by host | search NOT [inputlookup forwarders.csv | fields+ host]
| stats values(host) AS newhosts
modified the answer to reflect, indeed forgot to put the search after pie. this is the reason for your error, there is no NOT command
| tstats count where index = _* by host | search NOT [| inputlookup forwarders.csv | fields+ host] | stats values(host) AS new_hosts
There should be a pipe before inputlookup.
The current provided answers would work. This is how I would do it though:
| metadata type=hosts | eval daysSinceFirstTime = round((now() - firstTime)/86400, 2) | eval hoursSinceLastTime = round((now() - lastTime )/3600 , 2) | convert ctime(firstTime) as firstTime | convert ctime(lastTime) as lastTime | search daysSinceFirstTime < 1 OR (hoursSinceLastTime>24 AND hoursSinceLastTime<48)
In this example, you could run the alert search once a day to get all hosts that sent their first event to your indexer(s) in the past day and hosts that sent their last event to your indexer(s) between 24 and 48 hours ago. Run the search for all time as this is an extremely efficient command.
Refer here for the
metadata command documentation.