Splunk Search

Proper command_line format for perfdata log files

kultar
Engager

Hey All,

Just going through getting Splunk for Nagios installed and I followed the instructions as provided and all went well, except that on the Status Dashboard I don't get any results for Top 10 Service Notifications with status Warning/Critical.

What iv'e figured out is that splunk isn't parsing my log files and I believe it could possibly be my log files that are not being formulated properly.

Could someone verify that the following definitions are indeed correct?

# 'nagios-process-host-perfdata' command definition
define command{
        command_name    nagios-process-host-perfdata
        command_line    /usr/bin/printf "%b" "$TIMET$ src_host=\"$HOSTNAME$\" perfdata=\"HOSTPERFDATA\" hoststate=\"$HOSTSTATE$\" attempt=\"$HOSTATTEMPT$\" statetype=\"$HOSTSTATETYPE$\" executiontime=\"$HOSTEXECUTIONTIME$\" reason=\"$HOSTOUTPUT$\" result=\"$HOSTPERFDATA$\"\n" >> /opt/nagios/var/host-perfdata
        }

# 'nagios-process-service-perfdata' command definition
define command{
        command_name    nagios-process-service-perfdata
        command_line    /usr/bin/printf "%b" "$TIMET$ src_host=\"$HOSTNAME$\" perfdata=\"SERVICEPERFDATA\" name=\"$SERVICEDESC$\" severity=\"$SERVICESTATE$\" attempt=\"$SERVICEATTEMPT$\" statetype=\"$SERVICESTATETYPE$\" executiontime=\"$SERVICEEXECUTIONTIME$\" latency=\"$SERVICELATENCY$\" reason=\"$SERVICEOUTPUT$\" result=\"$SERVICEPERFDATA$\"\n" >> /opt/nagios/var/service-perfdata
        }

If I click the "Inspect..." button, the search it shows is

 search index="nagios" nagiosevent="SERVICE NOTIFICATION" statusnotification="WARNING" | dedup servicenamenotification hostnotification | top servicenamenotification limit="10" | fields + servicenamenotification count

But by just going through and doing a broad search on index=nagios, I do not have any nagiosevent="SERVICE NOTIFICATION" just, "SERVICE ALERT". Also, it appears as though my "statusnotification" is being parsed as "severity".

Is there a way I can adjust my config files to properly parse these nagios log files ? Any help would be greatly appreciated.

0 Karma

lukeh
Contributor

Hi 🙂

Those definitions are for the host and service performance logs respectively, however the field extraction for 'nagiosevent' is relevant only to nagios.log i.e. sourcetype=nagios

Please try this search over the last 30 days:

index="nagios" "SERVICE NOTIFICATION"

If some events are returned by Splunk, please cut & paste a few sample events here.

Which version of the following apps are you running in your environment:

Splunk

Splunk for Nagios

Nagios

MK Livestatus

All the best,

Luke 🙂

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...