Powershell script triggered from alert is not executing but I can see it triggered in python.log

neiljpeterson — Fri, 03 Oct 2014 00:26:17 GMT

My forehead is sore from banging it on my desk. Please help.

I cannot get scripts to run from an alert. The following is all the relevant info I have been using to trouble shoot this.

Implementation

To make it easy, I configured the alert to run "always" so the search term is irrelevant. The alert should be triggered every minute regardless. (I have also ran this with a search that I was manually triggering events in)

The powershell script

#!C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
"hello world" | Out-File -FilePath .\hello_ps.txt

The savedsearches.conf stanzas

[script_test_ps]
action.script = 1
action.script.filename = test.ps1
alert.digest_mode = True
alert.suppress = 0
alert.track = 0
auto_summarize.dispatch.earliest_time = -1d@h
cron_schedule = * * * * *
enableSched = 1
search = sourcetype=nothing

Verification

We know the search is running, we can see the search run and we can watch the script being called but it never actually executes.

From python.log

2014-10-02 18:19:03,171 Central Daylight Time INFO  runshellscript:188 - runshellscript: ['C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe', 'D:\\Splunk\\bin\\scripts\\test.ps1', '0', 'index=testing sourcetype=script_test', 'index=testing sourcetype=script_test', 'script_test_ps', 'Saved Search [script_test_ps] always(0)', 'https://splunkweb.domain.tld/app/search/@go?sid=scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935', '', 'D:\\Splunk\\var\\run\\splunk\\dispatch\\scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935\\results.csv.gz']

From scheduler.log

10-02-2014 18:19:03.889 -0500 INFO  SavedSplunker - savedsearch_id="neil.peterson;search;script_test_ps", user="neil.peterson", app="search", savedsearch_name="script_test_ps", status=success, digest_mode=1, scheduled_time=1412291940, dispatch_time=1412291942, run_time=0.562, result_count=0, alert_actions="script", sid="scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935", suppressed=0, thread_id="AlertNotifierWorker-1"

Troubleshooting

Powershell execution policy is unrestricted

PS D:\Splunk\bin\scripts> Get-ExecutionPolicy
Unrestricted

Splunkd is running as a service account

PS D:\Splunk\bin\scripts> Get-WmiObject win32_service | Where-Object {$_.name -like "splunk*"} | Select-Object name, startname

name                                                        startname
----                                                        ---------
Splunkd                                                     domain\svc.splunk
splunkweb                                                   domain\svc.splunk

The service account is in the Administrators groups

The troubleshooting steps from this wiki article: http://wiki.splunk.com/Community:TroubleshootingAlertScripts

Is my scheduled search running? YES. I see it in scheduler.log
Is my scheduled search generating the expected results? YES. I can watch the results come in on a real time search. I have also scheduled the alert to run "always" to make the search part of it irrelevant.
Is my alert action being triggered? YES. I have added email actions and I get those emails, as well as watching it in https://splunkweb.domain.tld/en-US/app/launcher/job_management?savedSearch=script_test
Is my alert script working? YES. I can run it from the command line, as the svc.splunk user, but when the alert triggers it, nothing happens.

Reference

I have been using the following documentation. I have read it and reread it.

http://docs.splunk.com/Documentation/Splunk/6.1.3/alert/ConfiguringScriptedAlerts

http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Setupalertactions#Run_a_script

http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/Savedsearchesconf

http://wiki.splunk.com/Community:TroubleshootingAlertScripts

I am out of rope on this one. Any help is appreciated. I have also done all of the above with a batch script as well with the same disappointing results.

I have been testing and troubleshooting this from lots of different angles. I tried to break everything down to the simplest example, but my copy pasta above may still have gotten mixed up. If so please point it out and I will verify my testing results.

EDIT: For posterity sake, I will explain the (very very very elementary and obvious) solution here. The script was writing out to $splunk_home\bin not $splunk_home\bin\scripts where the script was located. If you browse to the scripts location and run it, it obviously works, but when Splunk runs it, those relative paths do not exist from bin/

The other quirk I have noticed is that when Splunkd is running as LocalSystem, it can execute batch scripts but not powershell. When Splunk is running as a service account the powershell scripts work fine.

To wrap your ps1 up into a bat you should put the folllowing in the batch script.

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File .\location\ofyour\script.ps1

The above will look for $splunk_home\bin\location\ofyour\script.ps1 which is what we want.

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log

linu1988 — Fri, 03 Oct 2014 04:26:06 GMT

where do you expect the file to be created? give the absolute file path rather than the one you have mentioned. I have not tries to give a ps1 file in the script trigger path ever. So could you try with a bat file? just mention complete path to script and output file everywhere. Make the cron schedule to */1 * * * *. It's quite easy to test with the below in a .bat file

%SPLUNK_ARG_5% > e:\output.txt

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log

neiljpeterson — Fri, 03 Oct 2014 15:57:14 GMT

:hangs head in shame: The script was actually writing to $splunk_home\bin, not to $splunk_home\bin\scripts where the script was actually located. I found a bunch of text files from my testing in bin 😕

I had tested a hard path earlier, but I think I had other unresolved issues as well at that point so I errantly dismissed that as a solution.

Failure in basic troubleshooting I'm afraid 😐 I will blame it on a lack of sleep.

Thank you for your help! If you post it has an answer I can accept it.

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log

halr9000 — Fri, 03 Oct 2014 16:27:48 GMT

Converted to an answer for ya.

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log

halr9000 — Mon, 13 Jul 2015 18:51:28 GMT

Hey @neiljpeterson, if I'm understanding you correctly, the final solution uses a .bat file in action.script.filename, is that right? I just pinged docs team to add this to the official scripted alert docs, and wanted to make sure I had that detail right. Or were you only mentioning .bat because of the issues when running PowerShell as localsystem?

topic Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log in Getting Data In

Powershell script triggered from alert is not executing but I can see it triggered in python.log

Implementation

Verification

Troubleshooting

Reference

I am out of rope on this one. Any help is appreciated. I have also done all of the above with a batch script as well with the same disappointing results.

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log

Re: Powershell script triggered from alert is not executing but I can see it triggered in python.log