Deployment Architecture

Splunk App for Unix and Linux - Alerts don't fire

New Member

Hi,

I've installed the Splunk App for Unix and Linux on a Windows 2008 Server and Splunk Add-on for Unix and Linux on a Universal Forwarder on a Linux RedHat 5.5 Server.

I've configured the Splunk App for Unix and Linux as shown in the documentation and activated scripted inputs on the forwarder. The forwarder is sending the data correctly (I find them in the "os" index).

I've configured the Alert "Processes_Exceeds_by_Host" with a threshold of 10 processes, but I can't find any alerts in the Alerts dashboard of Splunk App for Unix and Linux.

What configuration I have missed?

Thanks

Apaz

Tags (2)
0 Karma

Explorer

Hi,

I fought with the same issue and solved it. Same setup. 1 Splunk Server with 1 Splunk Forwarder. The issue is the way the saved searches are configured in the Splunk App for Unix and Linux. Below is the relevant stanza in its default configuration.

[Processes_Exceeds_by_Host]
action.summary_index = 1
action.summary_index.marker = unix_aggregated_alerts
action.summary_index._name = unix_summary
alert.digest_mode = True
alert.expires = 1d
alert.suppress = 1
alert.suppress.period = 1m
alert.track = 1
auto_summarize.dispatch.earliest_time = -5m@m
cron_schedule = */5 * * * *
counttype = number of events
disabled = 1
dispatch.earliest_time = -5m@m
dispatch.latest_time = now
displayview = unix_flashtimeline
enableSched = 1
quantity = 1
relation = greater than
search = `Processes_Exceeds_by_Host("`_unix_alert_threshold_Processes_Exceeds_by_Host`")`

The issues is the combination of the quantity and relation setting. In plain english it's saying "if # of results returned is greater than 1 fire the alert". When you have ONE server it will never fire. Also, it would hide the fact that there is a server firing alerts.

The fix:

  1. copy $SPLUNK_HOME/etc/apps/splunk_app_for_nix/default/savedsearches.conf to $SPLUNK_HOME/etc/apps/splunk_app_for_nix/local/savedsearches.conf
  2. Edit $SPLUNK_HOME/etc/apps/splunk_app_for_nix/local/savedsearches.conf
  3. Change quantity under [Processes_Exceeds_by_Host] to 0

I know this is an old question, hopefully it helps someone else.

-Felix

0 Karma

Splunk Employee
Splunk Employee

@fenrisdacat I have followed the same steps as you mentioned but still not getting the alerts on Alerts Dashboard it is showing as I have added "quantity = 0" for each stanza but not getting alerts.
Thanks

0 Karma

Path Finder

@fenrisdacat, after making the changes in SPLUNKHOME/etc/apps/splunkappfornix/local/savedsearches.conf, do you leave the file in that location or do you move it back to SPLUNKHOME/etc/apps/splunkappfornix/default location? I just installed the app this week and running into the same issue. Thanks

0 Karma

Explorer

I encountered same problem. To me about Home, Metric, Host were OK. But for Alerts part. I set that in Setting with some threshold.
And then I pointed to back to Alerts part. Just thing I see is Alert not found

0 Karma