Why am I unable to execute Powershell or batch alert scripts in Windows?

[mbrownec]

Hello all,

I can't seem to get Powershell or batch script to "successfully" execute.

When I attempt to run a batch, I receive the following error in splunkd.log:

Error while executing script [Error 193] %1 is not a valid Win32 application

In regards to a powershell script:

If I attempt to run a ps1, even with #!c:\windows\system32\WindowsPowerShell\v1.0\powershell.exe as the first line, the exit code is seemed to be not 0.

Having changed/added the following to runshellscript.py:

• Line 223: beneath the subprocess.popen call under if mswindows: add: output = p.stdout.read()
• Line 231: then under if code!=0: modify results = splunk.Intersplunk.generateErrorResults("Script: " + str(output))

I saw an error with runshellscript,py improperly escaping the arguments. Having had a | in my splunk query, runshellscript.py allowed the | to be treated as a command line pipe, not as part of the query.

After removing the pipe from the splunk query (and restoring runshellscript.py to it's original form), powershell seems to exit with code 0, but my script does not successfully execute/do anything observable.

The scripts are as follows:

tester.bat:

@echo off
echo %SPLUNK_ARG_0% > "C:\program files\splunk\bin\scripts\testbat.out"
echo "bat started" >> "C:\Program Files\Splunk\bin\scripts\testbat.out"
c:\windows\system32\WindowsPowerShell\v1.0\powershell.exe -file "C:\Program Files\Splunk\bin\scripts\tester.ps1" %*
echo "bat finishing" >> "C:\Program Files\Splunk\bin\scripts\testbat.out"

tester.ps1

#!c:\windows\system32\WindowsPowerShell\v1.0\powershell.exe
#docs.splunk.com/Documentation/Splunk/6.5.0/Alert/Configuringscriptedalerts
$scriptname = $Args[0]
$numberofeventsreturned = $Args[1]
$searchterms = $Args[2]
$fqquerystring = $Args[3]
$nameofreport = $Args[4]
$alerttriggerreason = $Args[5]
$reportbrowserurl = $Args[6]
$gzippedresultsfile = $Args[8]
write-output "start" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
write-output $($args -join ";")  >> "C:\Program Files\Splunk\bin\scripts\test.csv"
write-output "done" >> "C:\Program Files\Splunk\bin\scripts\test.csv"

I see some older documentation on Windows script execution in the following locations:
http://wiki.splunk.com/Community:TroubleshootingAlertScripts
http://docs.splunk.com/Documentation/Splunk/6.5.1/Alert/Configuringscriptedalerts
https://answers.splunk.com/answers/171871/powershell-script-triggered-from-alert-is-not-exec.html <== wish it was this simple

However, none of the suggestions are of assistance, and it appears to be an issue directly related to how runshellscript,py is written.

Has anyone had any success executing Alert scripts on Windows with Splunk v6.4.4?

Thank you for your time,

Matt

[mbrownec · Accepted solution]

Here's the deal. Splunk support has confirmed that direct execution of powershell scripts is not supported by runshellscript.py.

The "workaround" (not really a workaround as you'll see in a second) is to use a BATch script wrapper.

Also, do not use positional arguments with BATch scripts, as there are escaping problems, use the set environmental variables. Also, you must nest in double-quotes if you use a | character in your search.

Here is a test bat script that produces some verbose output for review:

echo "%SPLUNK_ARG_0%" > "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "%SPLUNK_ARG_1%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "%SPLUNK_ARG_2%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "%SPLUNK_ARG_3%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "%SPLUNK_ARG_4%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "%SPLUNK_ARG_5%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "%SPLUNK_ARG_6%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "%SPLUNK_ARG_8%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo ---we are testing with two percent variables-- >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %%1 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo ---we are testing with one percent variables-- >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %1 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %2 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %3 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %4 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %5 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %6 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %7 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %8 >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo ---we are testing with one percent variables embedded in single-quotes embedded in double quotes, semi-color separated-- >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo "'%0';'%1';'%2';'%3';'%4';'%5';'%6';'%7';'%8'">> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo ---we are testing with asterisk-- >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo %* >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
echo ---execute powershell-- >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"
powershell.exe -file "C:\Program Files\Splunk\bin\scripts\tester.ps1" "%SPLUNK_ARG_0%";"%SPLUNK_ARG_1%";"%SPLUNK_ARG_2%";"%SPLUNK_ARG_3%";"%SPLUNK_ARG_4%";"%SPLUNK_ARG_5%";"%SPLUNK_ARG_6%";"%SPLUNK_ARG_8%" >> "C:\Program Files\Splunk\bin\scripts\testbat.txt" 2>&1 
echo Errorlevel = %errorlevel% >> "C:\Program Files\Splunk\bin\scripts\testbat.txt"

Here is the powershell script that's called within the above batch

$args > "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[0]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[0] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[1]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[1] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[2]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[2] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[3]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[3] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[4]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[4] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[5]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[5] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[6]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[6] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[7]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[7] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[8]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[8] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[9]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[9] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[10]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[10] >> "C:\Program Files\Splunk\bin\scripts\test.csv"
"--args[11]---------------" >> "C:\Program Files\Splunk\bin\scripts\test.csv"
$args[11] >> "C:\Program Files\Splunk\bin\scripts\test.csv"

Given the various issues I faced while passing the variables to the powershell script (escaping, etc), I found it best to rely on the powershell script to simply parse out a single argument... as is passed to the powershell script via the batch script call to powershell.exe.

Here is the contents of the text file generated by the batch just so you see how weird escaping is (or lack there of):

"C:\Program Files\Splunk\bin\scripts\tester.bat" 
"1" 
"host="DAMEWARESERVER" sourcetype="WinEventLog:Application" SourceName=dwmrcs EventCode=111 | table _time host User_ID" 
"host="DAMEWARESERVER" sourcetype="WinEventLog:Application" SourceName=dwmrcs EventCode=111 | table _time host User_ID" 
"Dameware Connections" 
"Saved Search [Dameware Connections] number of events(1)" 
"http://SPLUNKSERVER:8000/app/search/@go?sid=scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613" 
"C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613\results.csv.gz" 
---we are testing with two percent variables-- 
%1 
---we are testing with one percent variables-- 
1 
"host^=^'DAMEWARESERVER^'^ sourcetype^=^'WinEventLog:Application^'^ SourceName^=dwmrcs^ EventCode^=111^ ^|^ table^ _time^ host^ User_ID" 
"host^=^'DAMEWARESERVER^'^ sourcetype^=^'WinEventLog:Application^'^ SourceName^=dwmrcs^ EventCode^=111^ ^|^ table^ _time^ host^ User_ID" 
"Dameware^ Connections" 
"Saved^ Search^ ^[Dameware^ Connections^]^ number^ of^ events^(1^)" 
"http://SPLUNKSERVER:8000/app/search/@go?sid^=scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613" 
"" 
C:\Program 
---we are testing with one percent variables embedded in single-quotes embedded in double quotes, semi-color separated-- 
"'"C:\Program Files\Splunk\bin\scripts\tester.bat"';'1';'"host='DAMEWARESERVER' sourcetype='WinEventLog:Application' SourceName=dwmrcs EventCode=111 | table _time host User_ID"';'"host='DAMEWARESERVER' sourcetype='WinEventLog:Application' SourceName=dwmrcs EventCode=111 | table _time host User_ID"';'"Dameware Connections"';'"Saved Search [Dameware Connections] number of events(1)"';'"http://SPLUNKSERVER:8000/app/search/@go?sid=scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613"';'""';'C:\Program'"
---we are testing with asterisk-- 
1  "host^=^'DAMEWARESERVER^'^ sourcetype^=^'WinEventLog:Application^'^ SourceName^=dwmrcs^ EventCode^=111^ ^|^ table^ _time^ host^ User_ID"  "host^=^'DAMEWARESERVER^'^ sourcetype^=^'WinEventLog:Application^'^ SourceName^=dwmrcs^ EventCode^=111^ ^|^ table^ _time^ host^ User_ID"  "Dameware^ Connections"  "Saved^ Search^ ^[Dameware^ Connections^]^ number^ of^ events^(1^)"  "http://SPLUNKSERVER:8000/app/search/@go?sid^=scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613"  "" C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613\results.csv.gz 
---execute powershell-- 
Errorlevel = 0 

And the powershell output:

C:\Program Files\Splunk\bin\scripts\tester.bat;1;host=DAMEWARESERVER sourcetype=WinEventLog:Application SourceName=dwmrcs EventCode=111 | table _time host User_ID;host=DAMEWARESERVER sourcetype=WinEventLog:Application SourceName=dwmrcs EventCode=111 | table _time host User_ID;Dameware Connections;Saved Search [Dameware Connections] number of events(1);http://SPLUNKSERVER:8000/app/search/@go?sid=scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613;C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613\results.csv.gz
--args[0]---------------
C:\Program Files\Splunk\bin\scripts\tester.bat;1;host=DAMEWARESERVER sourcetype=WinEventLog:Application SourceName=dwmrcs EventCode=111 | table _time host User_ID;host=DAMEWARESERVER sourcetype=WinEventLog:Application SourceName=dwmrcs EventCode=111 | table _time host User_ID;Dameware Connections;Saved Search [Dameware Connections] number of events(1);http://SPLUNKSERVER:8000/app/search/@go?sid=scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613;C:\Program Files\Splunk\var\run\splunk\dispatch\scheduler__admin__search__RMD562ef57e39918d377_at_1481747700_13613\results.csv.gz
--args[1]---------------
--args[2]---------------
--args[3]---------------
--args[4]---------------
--args[5]---------------
--args[6]---------------
--args[7]---------------
--args[8]---------------
--args[9]---------------
--args[10]---------------
--args[11]---------------

Parse the semi-color separated $args[0] with -split or something else to determine the fields as described in the documentation: http://docs.splunk.com/Documentation/Splunk/6.5.1/Alert/Configuringscriptedalerts