We are planning to do some linux patching on some of our Prod servers and were thinking of setting some splunk alert which detects and reports any kind of anomalies in our logs after the patching is done. However, I have had varied results with anomalousvalues and anomalydetection. Currently I am using a combination of both. My concern however is, suppose we have 4 hosts under one sourcetype, and all the hosts restart after patching and this is logged in my logs, would splunk treat it as anomaly of each host or club them together and not treat it as an anomaly. I guess, the gist of my question is, can I have anomalydetection for each host without setting an alert for each individual host?
sourcetype="prod_gw" source="/opt/mqm/logs/err01.log" | anomalousvalue |anomalydetection is my current search for alert where sourcetype is a group of 4 hosts.
... View more
I have an alert to send us an email every time an app on the server is stopped. I am searching for a stopped pattern and generating the alert. However, there are times when we manually stop this app for maintenance and start it back up after the work is done. Now, this would generate a false alert since the app is stopped.
Is there anyway to avoid this? I mean, i only want to get alerted when the app is stopped and not restarted.
Here's an example:
I stop the app for maintenance at 1.00 PM. A app is stopped error is logged in the logs. I start it back at 1.10.
After multiple different lines in the logs, starting flow message is logged. However, in my current alert, since it's stopped in logged, i get alerted even though I have started the app back up. I am trying to avoid that.
My logs when the app is stopped has this line(my current search pattern).
Cannot process event as "app" is stopped
Multiple lines of different texts and words.....
Starting flow: app
So I should only get alerted if the starting flow line is not present in the logs in the set time period.
... View more