I am trying to determine if there is a way to search for the following scenario.
Bonded interface goes down and Splunk matches the search terms First search: "souretype=syslog earliest=-1h bond0: failed"
Once the first search terms are matched a second search based around the timeline of the first match is initiated but expanded by 30 minutes on each side to check for a manual interface shutdown, or a server restart. If EITHER search matches then ignore the search. If NEITHER secondary search matches then a bonded interface failure has accurred and a notification sent out.
Secondary search "modinfo bond error OR syslog-ng restart"
My initial thought was to leverage a subsearch but didn't know how I would be able to grab the host names of the first search, and use that in the second search to make sure the first search host sync's with the errors found in the second.
expand my search by 30 minutes each side of the initial matched search. Also, I need to pull the host or hosts that matches within the first search and use that in the second search to make sure the first search host sync's with the errors found in the second.
It might look something like this...but it failed.
sourcetype=syslog host= modinfo bond error OR syslog-ng restart earliest=-2h [search sourcetype=syslog earliest=-1h bond0: failed]
Try this:
sourcetype=syslog earliest=-1h bond0: failed | stats count earliest(_time) AS earliestTime latest(_time) AS latestTime | map search="modinfo bond error OR syslog-ng restart earliest=($earliestTime$ - 1800) latest=($latestTime$ + 1800)"