Hello,
Zenoss v.5 listens only on HTTPS for any incoming event.
'Create_event' function of this add-on doesn't seem to handle SSL, so HTTPS connection fails.
Any update/work-around for this?
thanks,
The current version of the app v2.0.2 is compatible with Zenoss 5.x
I've created a shell script for Splunk to run as an alert trigger action which will loop through events and post to the Zenoss api.
Note: you may have to customize for fields relevant to your splunk and zenoss environments respectively.
If this script is not working, verify that you have placed it in $SPLUNK_HOME/bin/scripts and have restarted splunk
Alerts should reference zenoss-alert-forward-wrapper.sh which enables logging and calls zenoss-alert-forward.sh
Template Search:
index=main
| head 1
| eval severity="INFO"
| eval resource=host_fqdn
| eval component=sourcetype+":"+source
| eval evclass="/App/Splunk"
| eval summary="Splunk to Zenoss test pulse"
| eval comment="--------------------------------------------------------------------------------"
| eval comment="valid values for severity: CRITICAL ERROR WARNING INFO DEBUG CLEAR"
| eval comment="resource must be host_fqdn if wanting to link to a host object in Zenoss or we may have to have a Zenoss script to do the conversion?"
| eval comment="component is currently set to soucetype + source, however can be customized. Check with your Zenoss admin"
| eval comment="evclass should always be /App/Splunk unless the Zenoss admin asks you to change it"
| eval comment="table command is order-sensitive i.e. last command in an alert sent to zenoss should always be | table severity resource component evclass summary"
| eval comment="--------------------------------------------------------------------------------"
| table severity resource component evclass summary
| fillnull value=missing_value
zenoss-alert-forward-wrapper.sh
#!/usr/bin/env sh
# export all vars so the child script will have access to them
# if this script is not working, verify that you have placed it in $SPLUNK_HOME/bin/scripts and have restarted splunk
export SEARCH_COUNT="$1" # $1 - Number of events returned
export SEARCH_TERMS="$2" # $2 - Search terms
export SEARCH_QUERY="$3" # $3 - Fully qualified query string
export SEARCH_NAME="$4" # $4 - Name of saved search
export SEARCH_REASON="$5" # $5 - Reason saved search triggered
export SEARCH_URL="$6" # $6 - URL/Permalink of saved search
export SEARCH_TAGS="$7" # $7 - Always empty as of 4.1
export SEARCH_PATH="$8" # $8 - Path to raw saved results in Splunk instance (advanced)
export RESULTS_GZ="$SEARCH_PATH"
SCRIPT_DIR=$SPLUNK_HOME/bin/scripts
LOG_FILE=$SPLUNK_HOME/var/log/splunk/zenoss-alert-forward.log
# execute script and log standard out and standard error
$SCRIPT_DIR/zenoss-alert-forward.sh 2>&1 > $LOG_FILE
zenoss-alert-forward.sh
#!/usr/bin/env sh
# zenoss-alert-forward.sh
# read through splunk results and generate an alert per splunk event to push to zenoss api via curl
# if this script is not working, verify that you have placed it in $SPLUNK_HOME/bin/scripts and have restarted splunk
# rob@jordan2000.com
# version 1.0.5
# created 8/23/2017
# updated 11/7/2018
# Parameters passed in from the alert.
# $1-$9 is the positional parameter list.
# --------------------------------------------------------------------------------
# *** CONFIG PARAMS
# --------------------------------------------------------------------------------
ZENOSS_API_USER="your_zenoss_api_user_here"
ZENOSS_API_PASS="your_zenoss_api_pass_here"
ZENOSS_URL="https://your.zenoss.api.url.here:443"
ZENOSS_API_PATH="/zport/dmd/evconsole_router"
# maximum number of events that can be sent to Zenoss per Splunk alert
# each Splunk event will be sent as individual alert
# anything over the event limit will be truncated
EVENT_LIMIT=100
# in most cases, you shouldn't have to change parameters below this line
ZENOSS_API_URL="${ZENOSS_URL}${ZENOSS_API_PATH}"
ZENOSS_MESSAGE_BEGIN="'{\"action\":\"EventsRouter\",\"method\":\"add_event\",\"data\":[{"
ZENOSS_MESSAGE_END="}],\"type\":\"rpc\",\"tid\":1}'"
SCRIPT_DIR="$SPLUNK_HOME/bin/scripts"
LOG_FILE="$SPLUNK_HOME/var/log/splunk/zenoss-alert-forward.log"
TMP_DIR="/tmp/zenoss-alert-forward.$$"
CURL_POST_SCRIPT="$TMP_DIR/zenoss-alert-forward-curl-post.$$.sh"
echo "START SCRIPT"
echo "--------------------------------------------------------------------------------"
/bin/date
# create tmp dir so that this script can run multiple copies in parallel without collision
mkdir $TMP_DIR
# --------------------------------------------------------------------------------
# *** DETERMINE RESULTS LOCATION AND UNZIP
# --------------------------------------------------------------------------------
RESULTS_FILE=`echo $RESULTS_GZ|awk -F".gz" '{print $1}'`
echo "RESULTS_GZ: $RESULTS_GZ"
echo "RESULTS_FILE: $RESULTS_FILE"
echo "*** cd $TMP_DIR"
cd $TMP_DIR
echo "*** cp $RESULTS_GZ $TMP_DIR"
cp $RESULTS_GZ $TMP_DIR
echo "*** gunzip $RESULTS_FILE"
gunzip $RESULTS_GZ
# --------------------------------------------------------------------------------
# *** echo SCRIPT VARS
# --------------------------------------------------------------------------------
echo "SCRIPT_DIR: $SCRIPT_DIR"
echo "LOG_FILE: $LOG_FILE"
echo "TMP_DIR: $TMP_DIR"
# --------------------------------------------------------------------------------
# *** echo ZENOSS VARS
# --------------------------------------------------------------------------------
#echo "ZENOSS_API_USER: $ZENOSS_API_USER"
#echo "ZENOSS_API_PASS: $ZENOSS_API_PASS"
echo "ZENOSS_URL: $ZENOSS_URL"
echo "ZENOSS_API_PATH: $ZENOSS_API_PATH"
echo "ZENOSS_API_URL: $ZENOSS_API_URL"
echo "EVENT_LIMIT: $EVENT_LIMIT"
echo "SCRIPT_DIR: $SCRIPT_DIR"
echo "LOG_FILE: $LOG_FILE"
echo "TMP_DIR: $TMP_DIR"
echo "ZENOSS_MESSAGE_BEGIN: $ZENOSS_MESSAGE_BEGIN"
echo "ZENOSS_MESSAGE_END: $ZENOSS_MESSAGE_END"
# --------------------------------------------------------------------------------
# *** echo DEFAULT SPLUNK VARS
# --------------------------------------------------------------------------------
echo "SEARCH_COUNT: $SEARCH_COUNT"
#echo "SEARCH_TERMS: $SEARCH_TERMS"
echo "SEARCH_QUERY: $SEARCH_QUERY"
echo "SEARCH_NAME: $SEARCH_NAME"
echo "SEARCH_REASON: $SEARCH_REASON"
echo "SEARCH_URL: $SEARCH_URL"
#echo "SEARCH_TAGS: $SEARCH_TAGS"
echo "SEARCH_PATH: $SEARCH_PATH"
# --------------------------------------------------------------------------------
# *** READ THROUGH RESULTS RECORDS, PARSE & FORWARD ALERTS TO ZENOSS USING CURL
# --------------------------------------------------------------------------------
# this allows us to grab addtional Splunk fields written out from a scheduled alert as a record in the table format
# note: your SPLUNK query must be in the following format: base search | table SEVERITY RESOURCE COMPONENT EVCLASS SUMMARY otherwise you will need to mod this script to use a new format/order
echo "*** cat $RESULTS_FILE"
# enable line below for additional debug
# cat $RESULTS_FILE
# determine field positions from results header
HEADER_RECORD=`cat $RESULTS_FILE | head -1 | sed -e 's/,,/,/g'`
FIELD_01_NAME=`echo $HEADER_RECORD|awk -F"," '{print $1}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_02_NAME=`echo $HEADER_RECORD|awk -F"," '{print $2}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_03_NAME=`echo $HEADER_RECORD|awk -F"," '{print $3}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_04_NAME=`echo $HEADER_RECORD|awk -F"," '{print $4}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_05_NAME=`echo $HEADER_RECORD|awk -F"," '{print $5}'|sed -e 's/^"//' -e 's/"$//'`
echo "HEADER_RECORD: $HEADER_RECORD"
echo "FIELD_01_NAME: $FIELD_01_NAME"
echo "FIELD_02_NAME: $FIELD_02_NAME"
echo "FIELD_03_NAME: $FIELD_03_NAME"
echo "FIELD_04_NAME: $FIELD_04_NAME"
echo "FIELD_05_NAME: $FIELD_05_NAME"
cat $RESULTS_FILE|grep -v "__"|head -$EVENT_LIMIT|while read RECORD
do
# read through results records, set values and strip off any double quotes
# skipping every other delimiter since splunk has null values between delimiters in the csv.gz it outputs
# ie. value1,,value2,,value3,,value4,,value5,
echo "RECORD: $RECORD"
# replace double commas with single commas since stats and top commands change the event format
RECORD=`echo $RECORD | sed -e 's/,,/,/g'`
echo "RECORD: $RECORD"
#SEVERITY=`echo $RECORD|awk -F"," '{print $1}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_01_VALUE=`echo $RECORD|awk -F"," '{print $1}'|sed -e 's/^"//' -e 's/"$//'`
#RESOURCE=`echo $RECORD|awk -F"," '{print $2}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_02_VALUE=`echo $RECORD|awk -F"," '{print $2}'|sed -e 's/^"//' -e 's/"$//'`
#COMPONENT=`echo $RECORD|awk -F"," '{print $3}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_03_VALUE=`echo $RECORD|awk -F"," '{print $3}'|sed -e 's/^"//' -e 's/"$//'`
#EVCLASS=`echo $RECORD|awk -F"," '{print $4}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_04_VALUE=`echo $RECORD|awk -F"," '{print $4}'|sed -e 's/^"//' -e 's/"$//'`
#SUMMARY=`echo $RECORD|awk -F"," '{print $5}'|sed -e 's/^"//' -e 's/"$//'`
FIELD_05_VALUE=`echo $RECORD|awk -F"," '{print $5}'|sed -e 's/^"//' -e 's/"$//'`
case $FIELD_01_NAME in
severity)
SEVERITY=$FIELD_01_VALUE
;;
resource)
RESOURCE=$FIELD_01_VALUE
;;
component)
COMPONENT=$FIELD_01_VALUE
;;
evclass)
EVCLASS=$FIELD_01_VALUE
;;
summary)
SUMMARY=$FIELD_01_VALUE
;;
*)
echo "ERROR: FIELD_01_NAME value of $FIELD_01_NAME not expected!"
;;
esac
case $FIELD_02_NAME in
severity)
SEVERITY=$FIELD_02_VALUE
;;
resource)
RESOURCE=$FIELD_02_VALUE
;;
component)
COMPONENT=$FIELD_02_VALUE
;;
evclass)
EVCLASS=$FIELD_02_VALUE
;;
summary)
SUMMARY=$FIELD_02_VALUE
;;
*)
echo "ERROR: FIELD_02_NAME value of $FIELD_02_NAME not expected!"
;;
esac
case $FIELD_03_NAME in
severity)
SEVERITY=$FIELD_03_VALUE
;;
resource)
RESOURCE=$FIELD_03_VALUE
;;
component)
COMPONENT=$FIELD_03_VALUE
;;
evclass)
EVCLASS=$FIELD_03_VALUE
;;
summary)
SUMMARY=$FIELD_03_VALUE
;;
*)
echo "ERROR: FIELD_03_NAME value of $FIELD_03_NAME not expected!"
;;
esac
case $FIELD_04_NAME in
severity)
SEVERITY=$FIELD_04_VALUE
;;
resource)
RESOURCE=$FIELD_04_VALUE
;;
component)
COMPONENT=$FIELD_04_VALUE
;;
evclass)
EVCLASS=$FIELD_04_VALUE
;;
summary)
SUMMARY=$FIELD_04_VALUE
;;
*)
echo "ERROR: FIELD_04_NAME value of $FIELD_04_NAME not expected!"
;;
esac
case $FIELD_05_NAME in
severity)
SEVERITY=$FIELD_05_VALUE
;;
resource)
RESOURCE=$FIELD_05_VALUE
;;
component)
COMPONENT=$FIELD_05_VALUE
;;
evclass)
EVCLASS=$FIELD_05_VALUE
;;
summary)
SUMMARY=$FIELD_05_VALUE
;;
*)
echo "ERROR: FIELD_05_NAME value of $FIELD_05_NAME not expected!"
;;
esac
# check for null values and set to "null"
if [ "$SEVERITY" = "" ]
then
SEVERITY="null"
fi
if [ "$RESOURCE" = "" ]
then
RESOURCE="null"
fi
if [ "$COMPONENT" = "" ]
then
COMPONENT="null"
fi
if [ "$EVCLASS" = "" ]
then
EVCLASS="null"
fi
if [ "$SUMMARY" = "" ]
then
SUMMARY="null"
fi
# generate MSG_ID
MSG_ID_DATE=`date '+%Y-%m-%d.%H.%M.%S'`
MSG_ID="$MSG_ID_DATE.$$"
# append MSG_ID to summary for tracing - uncomment line below if you want a unique msg_id to appear in the zenoss summary - makes it simple to trace the message back to the Splunk logs
#SUMMARY="$SUMMARY msg_id: $MSG_ID"
echo "MSG_ID: $MSG_ID"
echo "SEVERITY: $SEVERITY"
echo "RESOURCE: $RESOURCE"
echo "COMPONENT: $COMPONENT"
echo "EVCLASS: $EVCLASS"
echo "SUMMARY: $SUMMARY"
if [ "$EVCLASS" != "/App/Splunk" ]
then
echo "ERROR: EVCLASS not parsed correctly and/or not set to /App/Splunk - This will result in an invalid message being pased to Zenoss!"
fi
# NOTE: zenoss does not seem to like the field name resource, although the zenoss UI shows resource, so changed the field name to device before sending the message
#ZENOSS_MESSAGE_DATA="\"severity\":\"${SEVERITY}\",\"resource\":\"${RESOURCE}\",\"component\":\"${COMPONENT}\",\"evclass\":\"${EVCLASS}\",\"summary\":\"${SUMMARY}\",\"evclasskey\":\"\""
ZENOSS_MESSAGE_DATA="\"severity\":\"${SEVERITY}\",\"device\":\"${RESOURCE}\",\"component\":\"${COMPONENT}\",\"evclass\":\"${EVCLASS}\",\"summary\":\"${SUMMARY}\",\"evclasskey\":\"\""
echo "ZENOSS_MESSAGE_DATA: $ZENOSS_MESSAGE_DATA"
ZENOSS_MESSAGE="$ZENOSS_MESSAGE_BEGIN$ZENOSS_MESSAGE_DATA$ZENOSS_MESSAGE_END"
echo "ZENOSS_MESSAGE: $ZENOSS_MESSAGE"
#CURL_COMMAND="curl -v -k -1 -i -u $ZENOSS_API_USER:$ZENOSS_API_PASS -X POST -H \"Content-Type:application/json\" -d"
#CURL_COMMAND="curl -k -1 -i -u $ZENOSS_API_USER:$ZENOSS_API_PASS -X POST -H \"Content-Type:application/json\" -d"
CURL_COMMAND="curl -k -1 -u $ZENOSS_API_USER:$ZENOSS_API_PASS -X POST -H \"Content-Type:application/json\" -d"
echo "CURL_COMMAND: $CURL_COMMAND"
POST_COMMAND="$CURL_COMMAND $ZENOSS_MESSAGE $ZENOSS_API_URL"
echo "POST_COMMAND: $POST_COMMAND"
# had some trouble with the shell/curl properly passing arguments to zenoss, opted to build and run a dynamic script
# create script
echo "$POST_COMMAND" > $CURL_POST_SCRIPT
# set execute perms on script
chmod 700 $CURL_POST_SCRIPT
# execute script
$CURL_POST_SCRIPT 2>&1
# remove script after execution
rm -f $CURL_POST_SCRIPT
done
# --------------------------------------------------------------------------------
# *** CLEANUP
# --------------------------------------------------------------------------------
# remove tmp files generated by the script
echo
echo "*** rm -rf $TMP_DIR"
rm -rf $TMP_DIR
/bin/date
echo "--------------------------------------------------------------------------------"
echo "END SCRIPT"
exit