In our environment Nagios and Splunk are integrated. We configured an alert in Nagios monitoring tool which fetches data from Splunk but in Nagios monitoring tool, it is showing as "UNKNOWN - Error in Application name "wms" ".
The alert is configured in such that it is using the script check_splunk_savedsearch_value.sh and it is taking three arguments.
check_splunk_savedsearch_value.sh -a wms -s "WMOS - EW - Number of Allocation records" -w 1
[root@nagios server]# ./check_splunk_savedsearch_value.sh -a wms -s "WMOS - EW - Number of Allocation records" -w 1
UNKNOWN - no output returned from splunk.ce.corp|"wms:WMOS - EW - Number of Allocation records"=ERROR
When we ran the script in debugging mode, the following command is not returning any output.
[root@nagios server ]# /usr/bin/curl -s -k -u username:password https://splunk.ce.corp:8089/servicesNS/monitor/wms/search/jobs/export -d 'search=savedsearch %22WMOS%20%2d%20EW%20%2d%20Number%20of%20Allocation%20records%22' -d output_mode=csv|sed 1d
[root@nagios server]#
Saved search "WMOS - EW - Number of Allocation records" does exist on that search head.
When we click on the saved search, Following is the error it is showing on Web page.
Error in 'script': Getinfo probe failed for external search command 'dbquery'
The search job has failed due to an error. You may be able view the job in the Job Inspector.
Following is the search query.
| dbquery "wmsewprd" "select count(*) from alloc_invn_dtl where stat_code = 11 and (sysdate - mod_date_time) * 24 * 60 > 30"
When we remove | from the above search query it is showing events. Is the | in the above search query making difference ?
If No, what could be the problem.
dbquery "wmsewprd" "select count(*) from alloc_invn_dtl where stat_code = 11 and (sysdate - mod_date_time) * 24 * 60 > 30"
