topic Compare field values with field values from events before in Splunk Search

Compare field values with field values from events before

ckunath — Thu, 30 Mar 2017 13:46:53 GMT

Hello, I am currently trying to set up an alert in Splunk by checking my eventdata after events that contain a list of ID's.
If an event occurs which has a list of ID's like this:

ID_LIST=2345,7865,9876

I want to iterate through every ID in this list and check if the ID has been mentioned in any log ever before. If it can't be found in earlier logs, it should trigger an alert. If it appears in earlier logs, nothing should happen.

So right now I prepared my event by extracting all ID's from found event into a field. How can I check whether or not the ID has been mentioned before?

Re: Compare field values with field values from events before

woodcock — Thu, 30 Mar 2017 14:15:54 GMT

Assuming that you are running this alert every 5 minutes and 60 days is enough backtrack, like this:

index=foo sourcetype=bar earliest=-60d latest=now [search index=foo sourcetype=bar earliest=-5m latest=now | stats count BY ID | fields ID] | stats latest(_raw) count BY ID | search count=1

However, it would make better sense to store the birthday of each event in a lookup by running a search for All time just once like this:

index=foo sourcetype=bar | stats min(_time) AS birthday BY ID | outputlooup IDbirthdays

And then do the alert and update like this:

index=foo sourcetype=bar earliest=-5m latest=now | lookup IDbirthdays ID OUTPUT birthday | search NOT birthday="*" | stats min(_time) AS birthday latest(_raw) AS _raw BY ID | outputlookup append=true | rename _raw AS latest_raw

Re: Compare field values with field values from events before

ckunath — Thu, 30 Mar 2017 14:43:13 GMT

Thanks for your quick answer!
The only problem I have is right now is that I have to extract the ID's out of the list first, and store them in a field like this

eval splitArray=split(_raw,":")
| eval id_only=mvindex(splitArray,1)
| rex field=id_only mode=sed "s/RECEIVE//g"
| eval listID=split(id_only,",")

How can I do the comparison between the values of listID with the values of ID from the main search? In your example you can straight up filter by ID.
(I am sorry, I am still pretty inexperienced when it comes to advanced SPL queries)

Re: Compare field values with field values from events before

woodcock — Thu, 30 Mar 2017 15:06:57 GMT

You really should do a proper field extraction so that ID (or listID) is automatically extracted at search time as a multivalued field. That is the only way to go and that is a completely different question.