Alerting

Alert to check whether an ID was mentioned before

Communicator

Hello,

in my logfiles I am sometimes getting an event that looks like this:

finished_ids: 1,2,3

What I am trying to construct is an alert that goes off when an ID in that list was not mentioned in my log files ever before.
How exactly can I do this? I can eval a field containing the id's of that list, but how can I backtrack the IDs that are not there with it?

Thank you in advance.

0 Karma

Legend

Hi ckunath,
if your IDs are listable, you can put them in a lookup and then verify if they are present in a period using a search like this:

your_search
| stats count by ID
| append [ | inputlookup my_ids.csv | dedup ID | count=0 | table ID count]
| stats sum(count) AS Total by ID
| where Total=0

In this way IDs with Total=0 are the ones missed in that period.

Bye.
Giuseppe

0 Karma

Communicator

Hi giuseppe,
Is there perhaps a way to not use lookup as solution?

0 Karma

Legend

Instead of lookup you can use a search, but it's a limited check because you're not sure to check all IDs:
in this example I'm checking if the IDs of the last hour were present in the 24 hours before:

your_search earliest=-25h@h latest=-h@h
| stats count by ID
| append [ 
      your_search earliest=-h@h latest=now
     | dedup ID 
     | count=0 
     | table ID count ]
| stats sum(count) AS Total by ID
| where Total=0

If the problem is to manage the lookup, you could generate it automatically using a scheduled search (e.g. every hour or every night):

your_search earliest=-h@h latest=now
| dedup ID 
| count=0 
| table ID count

I usually prefer use the lookup.

Bye.
Giuseppe

0 Karma

SplunkTrust
SplunkTrust

Are you using the same event finishedids for cross verifying your historical IDs? Have you already extracted finishedids as multi-valued comma separated field?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

Communicator

Hi niketnilay,

I extracted the ids of the list event in a multivalued field (id = 1,2,3) with the name of my historical ids.
I'm not sure what you mean by cross verifying my historical ids with the finished_ids event exactly.

0 Karma

SplunkTrust
SplunkTrust

you can use your search and then table id and outlookup id: ... | table id | outputlookup id.csv
then search again and compare with lookup:

  ... your search for id| NOT [| inputlookup id.csv | fields+ id]
          | stats values(id) AS new_id
0 Karma

SplunkTrust
SplunkTrust

@adonio - you're missing a "put" from outputlookup. For a minute there, I thought I had learned a new command. 😉 Also, inputlookup needs the pipe before it, IIRC.

0 Karma

SplunkTrust
SplunkTrust

oh boy outlookup, i am taking off for the rest of the day.
thanks for that!

Communicator

Hi adonio,
is there a way to do this query without having to rely on lookups? Perhaps do a join with another search that searches for all existing ids? I somehow can't make my lookups work..

0 Karma