Unable to retrieve data from Splunk

[pratapa]

Hi,

In our environment Nagios and Splunk are integrated. We configured an alert in Nagios monitoring tool which fetches data from Splunk but in Nagios monitoring tool, it is showing as "UNKNOWN - Error in Application name "wms" ".

The alert is configured in such that it is using the script check_splunk_savedsearch_value.sh and it is taking three arguments.

check_splunk_savedsearch_value.sh -a wms -s "WMOS - EW - Number of Allocation records" -w 1

[root@nagios server]# ./check_splunk_savedsearch_value.sh -a wms -s "WMOS - EW - Number of Allocation records" -w 1
UNKNOWN - no output returned from splunk.ce.corp|"wms:WMOS - EW - Number of Allocation records"=ERROR

When we ran the script in debugging mode, the following command is not returning any output.

[root@nagios server ]# /usr/bin/curl -s -k -u username:password https://splunk.ce.corp:8089/servicesNS/monitor/wms/search/jobs/export -d 'search=savedsearch %22WMOS%20%2d%20EW%20%2d%20Number%20of%20Allocation%20records%22' -d output_mode=csv|sed 1d
[root@nagios server]#

What could be the reason.

We see that Splunk forwarder is not installed on Nagios Production server. Is Splunk forwarder needs to be installed on Nagios production server.

[nickhills]

Does the saved search "WMOS - EW - Number of Allocation records" exist on that search head, and does the user invoking the Rest API call have permission to view it?

[pratapa]

Yes, the saved search "WMOS - EW - Number of Allocation records" is existing. When we click on "Edit Permissions" under "More Info" we see that only monitor is having read permissions.

Following is the output of the saved search in Splunk GUI.

WMOS - EW - Number of Allocation records
WMOS - EW - Number of Allocation records

Error in 'script': Getinfo probe failed for external search command 'dbquery'
The search job has failed due to an error. You may be able view the job in the Job Inspector.