Proper command_line format for perfdata log files

[kultar]

Hey All,

Just going through getting Splunk for Nagios installed and I followed the instructions as provided and all went well, except that on the Status Dashboard I don't get any results for Top 10 Service Notifications with status Warning/Critical.

What iv'e figured out is that splunk isn't parsing my log files and I believe it could possibly be my log files that are not being formulated properly.

Could someone verify that the following definitions are indeed correct?

# 'nagios-process-host-perfdata' command definition
define command{
        command_name    nagios-process-host-perfdata
        command_line    /usr/bin/printf "%b" "$TIMET$ src_host=\"$HOSTNAME$\" perfdata=\"HOSTPERFDATA\" hoststate=\"$HOSTSTATE$\" attempt=\"$HOSTATTEMPT$\" statetype=\"$HOSTSTATETYPE$\" executiontime=\"$HOSTEXECUTIONTIME$\" reason=\"$HOSTOUTPUT$\" result=\"$HOSTPERFDATA$\"\n" >> /opt/nagios/var/host-perfdata
        }

# 'nagios-process-service-perfdata' command definition
define command{
        command_name    nagios-process-service-perfdata
        command_line    /usr/bin/printf "%b" "$TIMET$ src_host=\"$HOSTNAME$\" perfdata=\"SERVICEPERFDATA\" name=\"$SERVICEDESC$\" severity=\"$SERVICESTATE$\" attempt=\"$SERVICEATTEMPT$\" statetype=\"$SERVICESTATETYPE$\" executiontime=\"$SERVICEEXECUTIONTIME$\" latency=\"$SERVICELATENCY$\" reason=\"$SERVICEOUTPUT$\" result=\"$SERVICEPERFDATA$\"\n" >> /opt/nagios/var/service-perfdata
        }

If I click the "Inspect..." button, the search it shows is

 search index="nagios" nagiosevent="SERVICE NOTIFICATION" statusnotification="WARNING" | dedup servicenamenotification hostnotification | top servicenamenotification limit="10" | fields + servicenamenotification count

But by just going through and doing a broad search on index=nagios, I do not have any nagiosevent="SERVICE NOTIFICATION" just, "SERVICE ALERT". Also, it appears as though my "statusnotification" is being parsed as "severity".

Is there a way I can adjust my config files to properly parse these nagios log files ? Any help would be greatly appreciated.

[lukeh]

Hi 🙂

Those definitions are for the host and service performance logs respectively, however the field extraction for 'nagiosevent' is relevant only to nagios.log i.e. sourcetype=nagios

Please try this search over the last 30 days:

index="nagios" "SERVICE NOTIFICATION"

If some events are returned by Splunk, please cut & paste a few sample events here.

Which version of the following apps are you running in your environment:

Splunk

Splunk for Nagios

Nagios

MK Livestatus

All the best,

Luke 🙂