in my logfiles I am sometimes getting an event that looks like this:
What I am trying to construct is an alert that goes off when an ID in that list was not mentioned in my log files ever before.
How exactly can I do this? I can eval a field containing the id's of that list, but how can I backtrack the IDs that are not there with it?
Thank you in advance.
if your IDs are listable, you can put them in a lookup and then verify if they are present in a period using a search like this:
your_search | stats count by ID | append [ | inputlookup my_ids.csv | dedup ID | count=0 | table ID count] | stats sum(count) AS Total by ID | where Total=0
In this way IDs with Total=0 are the ones missed in that period.
Instead of lookup you can use a search, but it's a limited check because you're not sure to check all IDs:
in this example I'm checking if the IDs of the last hour were present in the 24 hours before:
your_search earliest=-25h@h latest=-h@h | stats count by ID | append [ your_search earliest=-h@h latest=now | dedup ID | count=0 | table ID count ] | stats sum(count) AS Total by ID | where Total=0
If the problem is to manage the lookup, you could generate it automatically using a scheduled search (e.g. every hour or every night):
your_search earliest=-h@h latest=now | dedup ID | count=0 | table ID count
I usually prefer use the lookup.
Are you using the same event finishedids for cross verifying your historical IDs? Have you already extracted finishedids as multi-valued comma separated field?
I extracted the ids of the list event in a multivalued field (id = 1,2,3) with the name of my historical ids.
I'm not sure what you mean by cross verifying my historical ids with the finished_ids event exactly.
you can use your search and then table id and outlookup id:
... | table id | outputlookup id.csv
then search again and compare with lookup:
... your search for id| NOT [| inputlookup id.csv | fields+ id] | stats values(id) AS new_id
is there a way to do this query without having to rely on lookups? Perhaps do a join with another search that searches for all existing ids? I somehow can't make my lookups work..