topic Re: Alert to check whether an ID was mentioned before in Alerting

Alert to check whether an ID was mentioned before

ckunath — Tue, 02 May 2017 09:25:12 GMT

Hello,

in my logfiles I am sometimes getting an event that looks like this:

finished_ids: 1,2,3

What I am trying to construct is an alert that goes off when an ID in that list was not mentioned in my log files ever before.
How exactly can I do this? I can eval a field containing the id's of that list, but how can I backtrack the IDs that are not there with it?

Thank you in advance.

Re: Alert to check whether an ID was mentioned before

niketn — Tue, 29 Sep 2020 13:54:41 GMT

Are you using the same event finished_ids for cross verifying your historical IDs? Have you already extracted finished_ids as multi-valued comma separated field?

Re: Alert to check whether an ID was mentioned before

ckunath — Tue, 02 May 2017 10:44:05 GMT

Hi niketnilay,

I extracted the ids of the list event in a multivalued field (id = 1,2,3) with the name of my historical ids.
I'm not sure what you mean by cross verifying my historical ids with the finished_ids event exactly.

Re: Alert to check whether an ID was mentioned before

adonio — Tue, 02 May 2017 12:09:05 GMT

you can use your search and then table id and outlookup id: ... | table id | outputlookup id.csv
then search again and compare with lookup:

  ... your search for id| NOT [| inputlookup id.csv | fields+ id]
          | stats values(id) AS new_id

Re: Alert to check whether an ID was mentioned before

DalJeanis — Tue, 02 May 2017 13:01:36 GMT

@adonio - you're missing a "put" from outputlookup. For a minute there, I thought I had learned a new command. 😉 Also, inputlookup needs the pipe before it, IIRC.

Re: Alert to check whether an ID was mentioned before

adonio — Tue, 02 May 2017 13:13:29 GMT

oh boy outlookup, i am taking off for the rest of the day.
thanks for that!

Re: Alert to check whether an ID was mentioned before

ckunath — Wed, 03 May 2017 11:57:42 GMT

Hi adonio,
is there a way to do this query without having to rely on lookups? Perhaps do a join with another search that searches for all existing ids? I somehow can't make my lookups work..

Re: Alert to check whether an ID was mentioned before

gcusello — Wed, 03 May 2017 12:16:16 GMT

Hi ckunath,
if your IDs are listable, you can put them in a lookup and then verify if they are present in a period using a search like this:

your_search
| stats count by ID
| append [ | inputlookup my_ids.csv | dedup ID | count=0 | table ID count]
| stats sum(count) AS Total by ID
| where Total=0

In this way IDs with Total=0 are the ones missed in that period.

Bye.
Giuseppe

Re: Alert to check whether an ID was mentioned before

ckunath — Wed, 03 May 2017 12:47:33 GMT

Hi giuseppe,
Is there perhaps a way to not use lookup as solution?

Re: Alert to check whether an ID was mentioned before

gcusello — Wed, 03 May 2017 13:20:36 GMT

Instead of lookup you can use a search, but it's a limited check because you're not sure to check all IDs:
in this example I'm checking if the IDs of the last hour were present in the 24 hours before:

your_search earliest=-25h@h latest=-h@h
| stats count by ID
| append [ 
      your_search earliest=-h@h latest=now
     | dedup ID 
     | count=0 
     | table ID count ]
| stats sum(count) AS Total by ID
| where Total=0

If the problem is to manage the lookup, you could generate it automatically using a scheduled search (e.g. every hour or every night):

your_search earliest=-h@h latest=now
| dedup ID 
| count=0 
| table ID count

I usually prefer use the lookup.

Bye.
Giuseppe