Splunk Search

Compare field values with field values from events before

ckunath
Communicator

Hello, I am currently trying to set up an alert in Splunk by checking my eventdata after events that contain a list of ID's.
If an event occurs which has a list of ID's like this:

ID_LIST=2345,7865,9876

I want to iterate through every ID in this list and check if the ID has been mentioned in any log ever before. If it can't be found in earlier logs, it should trigger an alert. If it appears in earlier logs, nothing should happen.

So right now I prepared my event by extracting all ID's from found event into a field. How can I check whether or not the ID has been mentioned before?

Tags (1)
0 Karma

woodcock
Esteemed Legend

Assuming that you are running this alert every 5 minutes and 60 days is enough backtrack, like this:

index=foo sourcetype=bar earliest=-60d latest=now [search index=foo sourcetype=bar earliest=-5m latest=now | stats count BY ID | fields ID] | stats latest(_raw) count BY ID | search count=1

However, it would make better sense to store the birthday of each event in a lookup by running a search for All time just once like this:

index=foo sourcetype=bar | stats min(_time) AS birthday BY ID | outputlooup IDbirthdays

And then do the alert and update like this:

index=foo sourcetype=bar earliest=-5m latest=now | lookup IDbirthdays ID OUTPUT birthday | search NOT birthday="*" | stats min(_time) AS birthday latest(_raw) AS _raw BY ID | outputlookup append=true | rename _raw AS latest_raw

ckunath
Communicator

Thanks for your quick answer!
The only problem I have is right now is that I have to extract the ID's out of the list first, and store them in a field like this

eval splitArray=split(_raw,":")
| eval id_only=mvindex(splitArray,1)
| rex field=id_only mode=sed "s/RECEIVE//g"
| eval listID=split(id_only,",")

How can I do the comparison between the values of listID with the values of ID from the main search? In your example you can straight up filter by ID.
(I am sorry, I am still pretty inexperienced when it comes to advanced SPL queries)

0 Karma

woodcock
Esteemed Legend

You really should do a proper field extraction so that ID (or listID) is automatically extracted at search time as a multivalued field. That is the only way to go and that is a completely different question.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...