Hi Splunkers,
We have realized our "First Time Seen Running Windows Service " Correlation search seen below has been giving alot of false positives. This correlation search came with Splunk ES Content Update.
index=wineventlog eventtype=wineventlog_system signature_id=7036
| rex field=Message "The (?<serviceName>[\w\s-]*) service entered the (?<action>\w*) state"
| where action="running"
| inputlookup append=t previously_seen_running_windows_services
| multireport
[| stats earliest(eval(coalesce(_time, firstTime))) as firstTime, latest(eval(coalesce(_time, lastTime))) as lastTime by serviceName
| outputlookup previously_seen_running_windows_services
| where fact=fiction]
[| eventstats earliest(eval(coalesce(_time, firstTime))) as firstTime, latest(eval(coalesce(_time, lastTime))) as lastTime by serviceName
| where firstTime >= relative_time(now(), "-60m@m") AND isnotnull(_time)
| stats values(dest) as dest by _time, serviceName]
| table _time, serviceName, dest
After inspecting every hostname with this alert we have come to conclusion that we have actually seen this service before on the hostname. The alarm actually looks only for the last sixty minutes of the hostname. Is this rule intented that way or is there a method to fix and get cleaner positive results ?
We know that any services ran before, and still is running even though the alarm keeps mentioning that the service was seen running for the first time.
Thanks, Regards .
I think the logic should be tweaked at the end to negate it at the end from the lookup file or It could be done at the beginning also. Am posting to negate it at the end
eventtype=wineventlog_system signature_id=7036
| rex field=Message "The (?[\w\s-]) service entered the (?\w) state" | where action="running" | inputlookup append=t previously_seen_running_windows_services | multireport [| stats earliest(eval(coalesce(_time, firstTime))) as firstTime, latest(eval(coalesce(_time, lastTime))) as lastTime by serviceName | outputlookup previously_seen_running_windows_services | where fact=fiction ] [| eventstats earliest(eval(coalesce(_time, firstTime))) as firstTime, latest(eval(coalesce(_time, lastTime))) as lastTime by serviceName | where firstTime >= relative_time(now(), "-60m@m") AND isnotnull(_time) | stats values(dest) as dest by _time, serviceName] | table _time, serviceName, dest | search NOT [| inputlookup previously_seen_running_windows_services | fields _time, serviceName, dest]
the CS is using a lookup (previously_seen_running_windows_services) to check the seen services. Every time search runs splunk get this lookup to compare and records whatever he finds new. Its Ok to look just into the last 60 min since every time the search runs you are also calculating the difference from what you have recorded.
Check if the lookup previously_seen_running_windows_services is been properly written into.