topic Re: Custom alert action is not working in Alerting

Custom alert action is not working

mbasharat — Sun, 20 Feb 2022 15:42:31 GMT

Hi,

Snapshot: I had some alerts with script actions. Alerts are simply if value A exceeds value B by 10 more e.g. value A=411 and value B=400 then trigger that alert and trigger the respective script alert is pointed to. Whenever alerts were triggered, specific shell scripts were triggered. Script were simply a shell command . Whenever those command/s ran, they fed alerts to event pipeline of another tool. It worked perfectly fine as designed until we upgraded Splunk from 7.x to 8.x. After upgrade, Script actions option is still available but does not work anymore since functionality has been deprecated.

Alternative: Suggested functionality to use is custom alert actions from Splunk. I have developed a TA using Splunk Add On builder. This allows users to create same alerts where Action section allows them to select the script of choice to attach with alert so whenever alert is triggered, script is trigged and event is initiated as before up tp event pipeline of another tool. This TA has a python script which simply call the shell scripts of choice if a condition is met for respective script.

Issue/s:

1. When same shell command that in the script/s is ran manually from one of the Search Heads command line, it works perfectly. This is for testing to make sure it works.

2. When same shell command that in the script/s is ran via Splunk Add On Builder @ validation section, it gives syntax error for "-" in the command. When I escaped it with \ using "\-", syntax error resolves and test runs fine with success message. However, event that is supposed to be injected into another tools event pipeline, that event does not come at all.

3. When same shell command that in the script/s is ran WITHOUT "\-options" flags, it works. This is just for testing to make sure the most basic functionality is working.

So what can I do to make this python script run the commands we want it to run to restore functionality which we had before via scripted alerts actions? I am suspecyting issue with Python. FYI, these scripts with shell commands and python script to trigger them are all in one package under same TA. Alerts are in a different application and are globally shared as always.

We are on Splunk 8.x. Python is upgraded via Splunk upgrade to 3.x. I am suspecting issue with Python engine and libraries. I did some libraries check and some libraries are still @ 2.7 even after the upgrade.

# encoding = utf-8

from __future__ import print_function
from future import standard_library
standard_library.install_aliases()
import requests
import sys,os
import json
import logging
import logging.handlers
import subprocess
from subprocess import Popen, PIPE

def process_event(helper, *args, **kwargs):

helper.set_log_level(helper.log_level)

dropdown_list = helper.get_param("dropdown_list")

helper.log_info("dropdown_list={}".format(dropdown_list))

helper.log_info("Alert action alert_action_test started.")

# Basic test command. Working and making it thru the event messaging queue

if dropdown_list == 'abcscript.sh':

execute=subprocess.run("/opt/OV/bin/opcmsg application=ABC object=ABC severity=Major msg_text=TESTTESTTEST, stdout=subprocess.PIPE, text=True)

# Working syntax wise without any errors in Add On builder but not making it thru the messaging queue as desired for custom alert actions. Note: there is an option Description below that I need to take out to make it work because there is a syntax error with it because python is not aking values with spaces. (\-option Description>>>>)

if dropdown_list == 'abcscript.sh':

execute=subprocess.run("/opt/OV/bin/opcmsg application=ABC object=ABC severity=Major msg_t=TEST \-option CIHint=ABC_Abstract_Node \-option ETIHint=ABCPortal:Major \-option Description='The rate of SendSubmission failures to successes exceeded the threshold in the JVM logs' \-option category=ABC_JVM \-option subcategory=SendSubmissionRate", stdout=subprocess.PIPE, text=True)

# There are 4 scripts in total so I have three more drop downs and 3 more commands respectively like above.

Thanks in-advance!!!

Re: Custom alert action is not working

PickleRick — Sun, 20 Feb 2022 16:44:26 GMT

You're calling subprocess.run incorrectly. See https://docs.python.org/3/library/subprocess.html#subprocess.run

Re: Custom alert action is not working

mbasharat — Sun, 20 Feb 2022 18:42:16 GMT

Hi PickleRick,

I am having difficulty getting it so can you please tell me what is incorrect and how it needs to be?

Thanks!

Re: Custom alert action is not working

PickleRick — Sun, 20 Feb 2022 19:16:05 GMT

Let me quote the important part:

args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.

Re: Custom alert action is not working

mbasharat — Sun, 20 Feb 2022 22:31:01 GMT

I have tried three variations below after reading. Still no luck.

# Method 1

if dropdown_list == 'msb_get_new_acks_submission.sh':

subprocess.run(["/opt/OV/bin/opcmsg", "application=ABC", "object=ABC", "severity=Major", "msg_t=TEST", "-option", "CIHint=ABC-A2A_Abstract_Node", "-option", "ETIHint=TESTABCPortal_ABC:Major", "-option", "Description='The rate of SendSubmission failures to successes exceeded the threshold in the ABC IEP A2A JVM logs'", "-option", "category=ABC_A2A_JVM", "-option", "subcategory=SendSubmissionRate"], stdout=subprocess.PIPE, text=True)

# Method 2

if dropdown_list == 'msb_get_new_acks_submission.sh':

execute=subprocess.run('/opt/OV/bin/opcmsg application=ABC object=ABC severity=Major msg_t=TEST -option CIHint=ABC-A2A_Abstract_Node -option ETIHint=TESTABCPortal_ABC:Major -option category=ABC_A2A_JVM -option subcategory=SendSubmissionRate', shell=True, stdout=subprocess.PIPE, text=True, check=True)

# Method 3, command is what I defined just to reduce the clutter.

if dropdown_list == 'msb_get_new_acks_submission.sh':

res = subprocess.call('command', shell=True)

Re: Custom alert action is not working

PickleRick — Mon, 21 Feb 2022 07:35:32 GMT

OK. Your first and second form seem to be relatively OK (they should indeed spawn the subprocess unless you pointed your script to a wrong path.

But if this is your whole script, and you counted on subprocess.run returining the output from the process to the calling splunk daemon, it won't work becaues you set stdout=subprocess.PIPE.

So either print directly from subprocess.run() (don't capture the output) - kinda ugly solution. Or capture and then print it yourself.

And most importantly for any such development - add some loging layer so you know what's happening inside your script!

Re: Custom alert action is not working

mbasharat — Mon, 21 Feb 2022 14:53:34 GMT

Logging is enabled in the code + TA @ DEBUG level from beginning. I do not see any errors/concerns when code executes. Scripts are sitting in the bin/scripts.

When you say, "So either print directly from subprocess.run() (don't capture the output) - kinda ugly solution. Or capture and then print it yourself." Can you correct the python and provide me the correct one pls? I am at the dead end after trying many things. Thanks 🙂

Re: Custom alert action is not working

PickleRick — Mon, 21 Feb 2022 16:55:46 GMT

OK. Let me show you what I mean.

I create a file named test.py with this inside:

#!/usr/bin/env python

import subprocess

print ("Don't capture")
a=subprocess.run(("ls"))
print ("Done")
print ("Capture")
a=subprocess.run(("ls"),capture_output=True)
print ("Done")
print ("Captured")
print (a)

If I run it, I get:

$ ./test.py 
Don't capture
test.py
Done
Capture
Done
Captured
CompletedProcess(args='ls', returncode=0, stdout=b'test.py\n', stderr=b'')

As you can see - the first call to subprocess.run() "bypassed" my python script and directly produced output on the stdout. The second one did not. But in the second case, the command's output was captured into a.stdout, from where I can use it - for example print by print(a.stdout).