Alerting

Alert to check whether an ID was mentioned before

ckunath
Communicator

Hello,

in my logfiles I am sometimes getting an event that looks like this:

finished_ids: 1,2,3

What I am trying to construct is an alert that goes off when an ID in that list was not mentioned in my log files ever before.
How exactly can I do this? I can eval a field containing the id's of that list, but how can I backtrack the IDs that are not there with it?

Thank you in advance.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi ckunath,
if your IDs are listable, you can put them in a lookup and then verify if they are present in a period using a search like this:

your_search
| stats count by ID
| append [ | inputlookup my_ids.csv | dedup ID | count=0 | table ID count]
| stats sum(count) AS Total by ID
| where Total=0

In this way IDs with Total=0 are the ones missed in that period.

Bye.
Giuseppe

0 Karma

ckunath
Communicator

Hi giuseppe,
Is there perhaps a way to not use lookup as solution?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Instead of lookup you can use a search, but it's a limited check because you're not sure to check all IDs:
in this example I'm checking if the IDs of the last hour were present in the 24 hours before:

your_search earliest=-25h@h latest=-h@h
| stats count by ID
| append [ 
      your_search earliest=-h@h latest=now
     | dedup ID 
     | count=0 
     | table ID count ]
| stats sum(count) AS Total by ID
| where Total=0

If the problem is to manage the lookup, you could generate it automatically using a scheduled search (e.g. every hour or every night):

your_search earliest=-h@h latest=now
| dedup ID 
| count=0 
| table ID count

I usually prefer use the lookup.

Bye.
Giuseppe

0 Karma

niketn
Legend

Are you using the same event finished_ids for cross verifying your historical IDs? Have you already extracted finished_ids as multi-valued comma separated field?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

ckunath
Communicator

Hi niketnilay,

I extracted the ids of the list event in a multivalued field (id = 1,2,3) with the name of my historical ids.
I'm not sure what you mean by cross verifying my historical ids with the finished_ids event exactly.

0 Karma

adonio
Ultra Champion

you can use your search and then table id and outlookup id: ... | table id | outputlookup id.csv
then search again and compare with lookup:

  ... your search for id| NOT [| inputlookup id.csv | fields+ id]
          | stats values(id) AS new_id
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

@adonio - you're missing a "put" from outputlookup. For a minute there, I thought I had learned a new command. 😉 Also, inputlookup needs the pipe before it, IIRC.

0 Karma

adonio
Ultra Champion

oh boy outlookup, i am taking off for the rest of the day.
thanks for that!

ckunath
Communicator

Hi adonio,
is there a way to do this query without having to rely on lookups? Perhaps do a join with another search that searches for all existing ids? I somehow can't make my lookups work..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...