You could port the script to power shell, unfortunately I don't have access to a windows host with send_nsca.exe. All the best, Luke. ... View more

You could port the script to power shell, unfortunately I don't have access to a windows host with solutions enabler. All the best, Luke. ... View more

Hi Norbert 🙂 I developed an app for that 🙂 You can use Splunk For Nagios to search Nagios downtime events, alerts and notifications and trend problems over time. Over 40 field extractions are included, as well as 8 Saved Searches, and Advanced Dashboards featuring recent Warning and Critical Alerts, as well as Integration with MK Livestatus and the ability to Schedule Saved Searches in Splunk to send alerts to Nagios: http://splunk-base.splunk.com/apps/22374/splunk-for-nagios All the best, Luke 🙂 ... View more

Your props.conf should already have the following entry to ensure that each event is indexed as a single line: /opt/splunk/etc/apps/SplunkForNagios/default/props.conf [nagioshostperf] SHOULD_LINEMERGE = false [nagiosserviceperf] SHOULD_LINEMERGE = false You could try restarting splunk, I often find that this can solve line breaking issues 🙂 All the best, Luke 🙂 ... View more

Replace queue = nullQueue with FORMAT = nullQueue ... View more

Hi Matt 🙂 The nagios.log file contains alerts and notifications etc but the performance data is logged to a separate file, either: "service-perfdata" to be ingested into splunk with a sourcetype of "nagiosserviceperf" or "splunk-nagios-perfdata" to be ingested into splunk with a sourcetype of "nagiosperfdata" Use the latter if using pnp4nagios 🙂 You'll need to update the pnp4nagios script to output the performance data to an additional log file and ingested the new file into splunk with a new sourcetype. This way the performance data log format does not need to change and your rrd graphs will continue to work. 1/ Update the 'Bulk Mode' section within "process_perfdata.pl" as follows: print_log( "reading $pdfile for bulk update", 2 ); open (SPLUNK, '>>/opt/nagios/var/splunk-nagios-perfdata'); open( PDFILE, "< $pdfile" ); my $count = 0; while (<PDFILE>) { $count++; print_log( "Processing Line $count", 2 ); my @LINE = split(/\t/); %ENV = (); # cleaning ENV foreach my $k (@LINE) { $k =~ /([A-Z 0-9_]+)::(.*)$/; $ENV{ 'NAGIOS_' . $1 } = $2 if ($2); } print SPLUNK "$_\n"; if ( $ENV{NAGIOS_SERVICEPERFDATA} || $ENV{NAGIOS_HOSTPERFDATA} ) { parse_env(); process_perfdata(); } else { print_log( "No Perfdata. Skipping line $count", 2 ); } } print_log( "$count Lines processed", 1 ); if ( unlink("$pdfile") == 1 ) { print_log( "$pdfile deleted", 1 ); } else { print_log( "Could not delete $pdfile:$!", 1 ); } } else { print_log( "ERROR: File $opt_b not found", 1 ); } close (SPLUNK); } Note: only the following three new lines should be added to your existing script: open (SPLUNK, '>>/opt/nagios/var/splunk-nagios-perfdata'); print SPLUNK "$_\n"; close (SPLUNK); Replace /opt/nagios with the relevant path for your installation 🙂 2/ Update "$SPLUNK_HOME/etc/apps/SplunkForNagios/default/props.conf" with the following new sourcetype: [nagiosperfdata] EXTRACT-datatype = DATATYPE::(?P<datatype>[^\t]*) EXTRACT-src_host = HOSTNAME::(?P<src_host>[^\t]*) EXTRACT-name = SERVICEDESC::(?P<name>[^\t]*) EXTRACT-result = SERVICEPERFDATA::(?P<result>[^\t]*) EXTRACT-process = SERVICECHECKCOMMAND::(?P<process>[^\t]*) EXTRACT-hoststate = HOSTSTATE::(?P<hoststate>[^\t]*) EXTRACT-hoststatetype = HOSTSTATETYPE::(?P<hoststatetype>[^\t]*) EXTRACT-state = SERVICESTATE::(?P<state>[^\t]*) EXTRACT-statetype = SERVICESTATETYPE::(?P<statetype>\w+) SHOULD_LINEMERGE = false TIME_PREFIX = TIMET:: 3/ Add the new file "splunk-nagios-perfdata" to be ingested into splunk with a sourcetype of "nagiosperfdata" 4/ Update the dashboards in "$SPLUNK_HOME/etc/apps/SplunkForNagios/default/data/ui/views" and change any occurance of sourcetype="nagiosserviceperf" to sourcetype="nagiosperfdata" All the best, Luke 🙂 P.S. The 'CURRENT SERVICE STATE' events are logged to nagios.log at midnight everyday, ie. as they are only logged just once per day they cannot be used for creating performance graphs, hence the requirement to ingest the performance data from the specific log file. ... View more

Hi Matt 🙂 I have updated the doco because it was too vague and misleading, here are the new instructions: Using your favourite xml editor, update these dashboards with the relevant label/key names for the specific performance data that are in use in your nagios environment. eg. change avg(CpuSystem) to avg(cpu_system) if your performance data for CPU Usage is labelled differently. Dashboard location: $SPLUNK_HOME/etc/apps/SplunkForNagios/default/data/ui/views ie. you don't have to change the service_description for any of your nagios checks, you just need to update the dashboards in Splunk for Nagios with your specific performance data label/key names. Feel free to provide examples of your nagios performance data if you would like further assistance and I can recommend the relevant updates to make to your dashboards. All the best, Luke 🙂 ... View more

Hi 🙂 The hostname list in the Alerts Dashboard is populated by the following splunk search: earliest=-24h index="nagios" nagiosevent="CURRENT HOST STATE" | rex ".+CURRENT HOST STATE\: (?P<device>[^;]*)(?=;)"| stats count by device Could you please run that search and check if you have a list of devices (aka hostnames). If the above search does not return any results, please run the following search to confirm that your nagios.log file is being indexed by splunk with the correct sourcetype: earliest=-24h index="nagios" sourcetype=nagios Note: the nagios log contains events labelled "CURRENT HOST STATE" which are generated by default by Nagios at midnight every day. I also noticed that you have not changed the template in your nagios installation for the performance data, ie. the sample log data you included does not contain key=value pairs that Splunk for Nagios is expecting. Please follow the instructions entitled "Nagios Configuration (REQUIRED)" to fix the log format of the performance data: http://splunk-base.splunk.com/apps/22374/splunk-for-nagios Note: if you are using pnp4nagios or similar to generate rrd graphs then changing the log format of the performance data as above may break your rrd graphs. I am currently testing a methodology to create a new sourcetype so that both pnp4nagios and Splunk for Nagios can be used at the same time. Watch this space 🙂 All the best, Luke 🙂 ... View more

Please ensure that your Splunk server can talk to MK Livestatus by executing a simple netcat command, for example: root@homer:/opt/splunk/etc/apps/SplunkForNagios/bin# nc 10.10.10.10 6557 < nagios-hosts name,address,alias,hard_state bart,10.10.10.101,web server,0 lisa,10.10.10.102,database server,0 Note: replace 10.10.10.10 with the ip address of your nagios server running MK Livestatus If you don't get a result using netcat then you should look at MK Livestatus - ensure that the ip address of your splunk server is listed next to "only_from" in /etc/xinetd.d/livestatus on your nagios server. It is possible to update a python script to log errors, please refer to the following splunk answers for assistance: http://splunk-base.splunk.com/answers/30535/any-advice-for-troubleshooting-scripted-lookups http://splunk-base.splunk.com/answers/10283/python-scripted-lookup-doesnt-produce-any-results All the best, Luke 🙂 ... View more

I have updated this answer with the relevant field extraction 🙂 ... View more

That is good news Vishal, and you're welcome. Luke 🙂 ... View more

Please refer to this splunk answer: http://splunk-base.splunk.com/answers/32492/issue-with-livestatus-splunkintersplunkgetorganizedresults-never-returns-a-value All the best, Luke 🙂 ... View more

Hi Vishal 🙂 Glad you got the dashboard working 🙂 It is ok to use any valid hostname from your nagios configuration as the underlying lookup scripts require that the search result contains a field called src_host before performing the relevant MK Livestatus lookups. ie. the dashboard is populated by 8 different python lookup scripts and each of them are executed after a successfully completed splunk search that contains src_host in the results. So as long as you use a src_host that exists in the nagios logs which are indexed by splunk, the lookup scripts will perform their specific job. Luke 🙂 ... View more

Change the device name to any hostname in your nagios configuration. I would recommend that you choose the name of a host/device that is always up, eg. router/switch/server. All the best, Luke 🙂 ... View more

not very much 🙂 the sysstat log is approx 6MB per day and the EMCcapacity log is only approx 90KB per day... All the best, Luke 🙂 ... View more

I have released two new storage specific apps on Splunk Base: Splunk for Isilon: http://splunk-base.splunk.com/apps/32975/splunk-for-isilon Splunk for Symmetrix (VMAX supported): http://splunk-base.splunk.com/apps/32976/splunk-for-symmetrix Please feel free to ask if you have any questions or queries 🙂 Luke. ... View more

Please ensure that you rename the "name" value in the dashboard to the relevant name of your nagios plugin, ie. it should be the same as your service_description in nagios. Example plugin output from "check_mem.pl" is as follows: TOTAL=12167908KB;;;; USED=2661344KB;;;; FREE=9506564KB;;;; CACHES=9207732KB;;;; Note: the Splunk for Nagios compatible Memory plugin is available here: check_mem.pl: http://exchange.nagios.org/directory/Plugins/System-Metrics/Memory/check_mem-2Epl/details All the best, Luke 🙂 ... View more

The python scripts that are included in Splunk for Nagios will only work from within the app, ie. they won't work from the command line. Is the "Livestatus Dashboard" working? If not, you must edit NagiosLivestatus.xml and change the "src_host" name to a relevant device name in nagios. Hope this helps, Luke 🙂 ... View more

It may be possible that the data is being indexed in the default index called main, instead of the required destination index called nagios. If you run the following search in Splunk for Nagios are there any results: index=nagios If not, try the following search and note the name of the index listed in the field picker at left: source=/var/log/nagios3/nagios.log If the results are not in the nagios index, please edit $SPLUNK_HOME/etc/apps/search/local/inputs.conf on the Nagios server and add the following key/value pair: index = nagios then restart the Splunk UF agent: ./splunk restart Note: please ensure that the nagios index is owned by the splunk user, if not please fix the ownership and restart splunk on your splunk server. All the best, Luke 🙂 ... View more

Hurrican Labs released a python script for nagios monitoring of your splunk license usage, and I have included the script and a fresh dashboard in the latest version of Splunk for Nagios: http://splunk-base.splunk.com/apps/22374/splunk-for-nagios All the best, Luke 🙂 ... View more

Good to hear Scott, please feel free to accept my answer 🙂 ... View more

I have written an app called 'Splunk for Nagios' which will do exactly what you're after 🙂 http://splunk-base.splunk.com/apps/22374/splunk-for-nagios Essentially, you ingest the nagios log file into Splunk which gives you the ability to see all of your nagios events, including the output of your nagios plugins 🙂 All the best, Luke 🙂 ... View more