Hello,
in my logfiles I am sometimes getting an event that looks like this:
finished_ids: 1,2,3
What I am trying to construct is an alert that goes off when an ID in that list was not mentioned in my log files ever before.
How exactly can I do this? I can eval a field containing the id's of that list, but how can I backtrack the IDs that are not there with it?
Thank you in advance.
Hi ckunath,
if your IDs are listable, you can put them in a lookup and then verify if they are present in a period using a search like this:
your_search
| stats count by ID
| append [ | inputlookup my_ids.csv | dedup ID | count=0 | table ID count]
| stats sum(count) AS Total by ID
| where Total=0
In this way IDs with Total=0 are the ones missed in that period.
Bye.
Giuseppe
Hi giuseppe,
Is there perhaps a way to not use lookup as solution?
Instead of lookup you can use a search, but it's a limited check because you're not sure to check all IDs:
in this example I'm checking if the IDs of the last hour were present in the 24 hours before:
your_search earliest=-25h@h latest=-h@h
| stats count by ID
| append [
your_search earliest=-h@h latest=now
| dedup ID
| count=0
| table ID count ]
| stats sum(count) AS Total by ID
| where Total=0
If the problem is to manage the lookup, you could generate it automatically using a scheduled search (e.g. every hour or every night):
your_search earliest=-h@h latest=now
| dedup ID
| count=0
| table ID count
I usually prefer use the lookup.
Bye.
Giuseppe
Are you using the same event finished_ids for cross verifying your historical IDs? Have you already extracted finished_ids as multi-valued comma separated field?
Hi niketnilay,
I extracted the ids of the list event in a multivalued field (id = 1,2,3) with the name of my historical ids.
I'm not sure what you mean by cross verifying my historical ids with the finished_ids event exactly.
you can use your search and then table id and outlookup id: ... | table id | outputlookup id.csv
then search again and compare with lookup:
... your search for id| NOT [| inputlookup id.csv | fields+ id]
| stats values(id) AS new_id
@adonio - you're missing a "put" from outputlookup
. For a minute there, I thought I had learned a new command. 😉 Also, inputlookup
needs the pipe before it, IIRC.
oh boy outlookup, i am taking off for the rest of the day.
thanks for that!
Hi adonio,
is there a way to do this query without having to rely on lookups? Perhaps do a join with another search that searches for all existing ids? I somehow can't make my lookups work..