I have a scheduled savedsearch that may return a result such as this
_time, host, _raw
In this example, the content of <some message> causes an alert to fire, which is what I expect.
Now, assume that a new event occurs and the next scheduled search returns this (changes in bold):
Problem: The next scheduled search will return the entire list (5 events) and thus trigger an alert containing these 5 events. However, 3 of these events were contained in a previous alert and are thus superfluous.
Desired outcome: The new alert should only be triggered based on the two "new" events (in bold)
What I have tried: Set trigger type to "for each event" and suppress for fields _time and host because I would assume that the combination of _time and host will uniquely identify the event to suppress
I also tried to learn about dynamic input lookups, but the documentation seems to be lost / unavailable (http://wiki.splunk.com/Dynamically_Editing_Lookup_Tables)
probably the time frame you're using in your alert is greather than the scheduling period, so there's an overlapping of events.
The only solution, for my knowledge, is to adapt the time frame to the scheduling period:
if your alert is scheduled every 5 minutes, you have to use five minutes as time frame,
it's also better to use the start of the minute to avoid overlapping: e.g. for the last 5 minutes, use as timeframe earliest=-5m@m latest=@m.
Your assumption about the time frame is indeed correct.
I have to search within a timeframe of the last 30 days. This is a requirement because I may only receive the data in an asynchronous manner (the most recent event in a new file might already be a day old, or even older, when I receive it)
This leads to potential problems if I want to equal the scheduling period with the time frame, as you suggest.
adapt my approach to your search:
if there's an event_id to identify events
<your_search> | stats count BY event_id | where count>10 | search NOT [ search index=summary_alerts | fields event_id ] | collect index=summary alerts
in this way, you run your search and check the condition.
Then after the check, you filter results discarding the already indexed event_ids and you add to the summary index only the new ones.
Before going over to summary indeces, I would like to implement this using lookups instead.
The Logic is the same as with summary indices:
I cannot however get the inputlookup in a subsearch to work. Here is a very basic example:
Note: source_file is a field that I extract in props.conf
index = myIndex eventtype = myEventtype | fields _time, host, source_file | search NOT [ | inputlookup known-events.csv | fields _time, host, source_file ]
I have added a single line to known-events.csv (for testing purposes), so I would expect that the number of results for "myIndex" and "myEventtype" would decrease by one, but I am getting either the same results as before or none at all. I have confirmed that the single line in known-events.csv is acutally there.
don' use the filter in a following statement, puy it in the main search so your search will be faster:
index=myIndex eventtype=myEventtype NOT [ | inputlookup known-events.csv | fields _time, host, source_file ]
the most important think is that the field names in subsearch will be the same of the main search.
Thank you for your continued help.
I must be doing something fundamentally wrong. If I run the search as you describe it, it returns zero results.
To adress your point and make sure that the field names are the exact same, I tried this:
index = myIndex eventtype = myEventtype | fields _time, host, source_file NOT [| inputlookup known-events.csv | fields _time, host, source_file ]
This gives the following error:
Error in 'fields' command: Invalid argument: 'source_file=some_file_name-[some-host_name].txt'
I am not quite sure what to make of this