Can't get value of job.resultCount when using Custom Alert Action

[rune_hellem]

How the alert is defined

I have created a custom alert action after following documentation found here http://docs.splunk.com/Documentation/Splunk/6.6.0/AdvancedDev/ModAlertsIntro, my alert is defined like this

[nimsoft]
is_custom = 1
label = Nimsoft Custom Alert Action
icon_path = action.png
payload_format = json
disabled = 0
alert.execute.cmd = powershell.path
alert.execute.cmd.arg.0 = -NoProfile
alert.execute.cmd.arg.1 = -f
alert.execute.cmd.arg.2 = $SPLUNK_HOME\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1
alert.execute.cmd.arg.3 = --execute
param.result_count = $job.resultCount$
param.search_query = $job.search$

Problem description

The above alert is working almost just fine. Using Powershell I am able to get hold of both the payload and the command line arguments, script output is like this

[10/19/2017 8:23 AM]: Now loop all arguments
[10/19/2017 8:23 AM]: Arg 0: --execute
[10/19/2017 8:23 AM]: Settings are: @{app=klp_nimsoft_custom_alerts; owner=admin; results_file=D:\splunk\var\run\splunk\dispatch\scheduler__admin_...__TestAlarm_at_1508394180_23089\per_result_alert\tmp_24.csv.gz; results_link=http://SplunkSearch:80/app/klp_nimsoft_custom_alerts/search?q=%7Cloadjob%20scheduler__admin_....w__TestAlarm_at_....&earliest=0&latest=now; search_uri=/servicesNS/nobody/klp_nimsoft_custom_alerts/saved/searches/TestAlarm; server_host=SPLUNKSEARCH; server_uri=https://127.0.0.1:8089; session_key=iwb0t_....; sid=scheduler__admin_...__TestAlarm_at_1508394180_23089; search_name=TestAlarm; configuration=; result=}
[10/19/2017 8:23 AM]: All done

But problem is that in order to really do the magic I need to to with the proper script, I need to know the number of events found by the alert, that is

  param.result_count = $job.resultCount$

But the value is nowhere to be found, at least it can be found where I expect it to be found, so maybe I need to look into other places. Anyone able to see what's wrong here?

The Powershell script testArguments.ps1

It might be that someone is curious about how the script looks like as well, so here goes

<#
.Synopsis
Script used to verify Splunk alerts, write to a log file both command line argumenst and stdin (payload)

.Description
Powershell -File "D:\Splunk\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1"

#>


<#
    Get current timestamp, used when writing to logfile
#>
function Get-TimeStamp {

    $timeStamp = "[" + (Get-Date).ToShortDateString() + " " + ((Get-Date).ToShortTimeString()) + "]"

    Return $timeStamp

}

# The logfile
$fileName = "d:\temp\arguments_test_updated.log"

# If exist, remove
If (Test-Path $fileName) {
    Remove-Item $fileName
}

# Start printing all argv's
$msg = (Get-TimeStamp) + ": Now loop all arguments"
write-host  $msg
Add-Content $fileName $msg
for ( $i = 0; $i -lt $args.count; $i++ ) {
    $msg = (Get-TimeStamp) + ": Arg $($i): $($args[$i])"
    write-host $msg
    Add-Content $fileName $msg

}

# Print stdin
# https://stackoverflow.com/questions/44695956/what-is-powershells-equivalent-to-pythons-sys-stdin-rea...
$settings = $input | Out-String | ConvertFrom-Json
$msg = (Get-TimeStamp) + ": Settings are: " + $settings
write-host $msg
Add-Content $fileName $msg
$msg = (Get-TimeStamp) +  ": All done"
Write-Host $msg
Add-Content $fileName $msg

[rune_hellem · Accepted solution]

Ref. my comment from the 13'th of November I did create a support ticket. The ticket is now closed with the following conclusion

Thank you for the update. I understand that you got this working with python and are going to use this instead of Powershell. If you are moving to a clustered environment then we would definitely recommend going with a Linux solution.
I will close down this matter but will follow up with our engineering team with regards to using powershell.

(Blockquote is very bold these days...)

So for now I will leave the scripted alerts as they are using Powershell and sometime in the not so distant future I will convert to using Custom Alerts with Python instead. This will make the move from Windows to Linux less painfull as well if we are to cluster our Splunk deployment.

[jkat54]

According to this page they add param.trigger_reason for backwards compatibility. Did you try that?

http://docs.splunk.com/Documentation/Splunk/7.0.1/AdvancedDev/CustomAlertConvertScripted

alert_actions.conf.spec
[slackcustomalert]

param.trigger_reason = <string>
* Provided for backwards compatibility with scripted alerts

savedsearches.conf.spec

action.slackcustomalert.param.trigger_reason = <string>
* Provided for backwards compatibility with scripted alerts

[kamlesh_vaghela]

Hi

Have tried to pass hardcode value in conf file??

like..
param.result_count = 100
param.search_query = | stats count

Does payload populate as expected?
Thanks

[rune_hellem]

Did just now try

param.result_count = 67
param.search_query = | stats count

First /debug/refresh, then restart. Still

 search_name=TestAlarm; configuration=; result=}

To be all sure, did search for result_count in all files in folder

D:\splunk\var\run\splunk\dispatch\scheduler__admin_a2xwX25pbXNvZnRfY3VzdG9tX2FsZXJ0cw__TestAlarm_at_1509009960_21

Found nothing. Did also check the results.csv.gz and it does have results.

[kamlesh_vaghela]

can you share your savedsearch.conf configuration for this particular alert?

[rune_hellem]

[TestAlarm]
action.email.useNSSubject = 1
action.logevent = 1
action.logevent.param.event = nimsoftCustomAlert=true antall=$result.count$ antallJob=$job.resultCount$
action.logevent.param.host = splunksearch
action.logevent.param.index = filenetprod
action.nimsoft = 1
action.nimsoft_100_filenet_error = 1
alert.digest_mode = 0
alert.suppress = 0
alert.track = 0
counttype = number of events
cron_schedule = * * * * *
dispatch.earliest_time = -2m
dispatch.latest_time = now
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = klp_nimsoft_custom_alerts
request.ui_dispatch_view = search
search = index=filenetprod error

The action logevent creates an event of type generic_single_line which is logged like this
nimsoftCustomAlert=true antall= antallJob=44

[kamlesh_vaghela]

Hi

can please add below two entry in [TestAlarm]

action.nimsoft.param.result_count = $job.resultCount$
action.nimsoft.param.search_query = $job.search$

Meanwhile can you please check any error during executing of search?

index=_internal component=SavedSplunker savedsearch_name="TestAlarm"

index=_internal component=script command=runshellscript

[rune_hellem]

Hmm, hard nail this one...

Did add to savesearches.conf

action.nimsoft.param.result_count = $job.resultCount$
action.nimsoft.param.search_query = $job.search$

Restarted Splunk, still same

...search_name=TestAlarm; configuration=; result=}

Checking internal log, only info messages

10-26-2017 14:49:29.685 +0200 INFO  SavedSplunker - savedsearch_id="nobody;klp_nimsoft_custom_alerts;TestAlarm", search_type="scheduled", user="admin", app="klp_nimsoft_custom_alerts", savedsearch_name="TestAlarm", priority=default, status=success, digest_mode=0, scheduled_time=1509022140, window_time=0, dispatch_time=1509022141, run_time=0.875, result_count=23, alert_actions="logevent,nimsoft", sid="scheduler__admin_a2xwX25pbXNvZnRfY3VzdG9tX2FsZXJ0cw__TestAlarm_at_1509022140_14", suppressed=0, fired=23, skipped=0, action_time_ms=26641, thread_id="AlertNotifierWorker-0", message=""

and

index=_internal component=ScriptRunner

actually does return

10-26-2017 14:45:54.463 +0200 ERROR ScriptRunner - Couldn't start child process. script="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -f D:\splunk\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1 --execute"

But it seems to be a false positive, since the log file is being updated.

[kamlesh_vaghela]

mmm,

Is below is another alert action or something in [TestAlarm] ?? Can we comment It?
like..

#action.nimsoft_100_filenet_error = 1

Apart from it, can we validate configuration by executing btool command??

YOUR_SPLUNK_PATH/bin/splunk btool check

[rune_hellem]

Did clean up the alert, now it is defined like this

[TestAlarm]
action.email.useNSSubject = 1
action.nimsoft = 1
action.nimsoft.param.result_count = $job.resultCount$
action.nimsoft.param.search_query = $job.search$
alert.digest_mode = 0
alert.suppress = 0
alert.track = 0
counttype = number of events
cron_schedule = * * * * *
dispatch.earliest_time = -2m
dispatch.latest_time = now
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = klp_nimsoft_custom_alerts
request.ui_dispatch_view = search
search = index=filenetprod error

Restarted the server to be all sure, but still no help. Nothing in configuration yet.
Splunk btool check
does not show any issuew with the app.

[kamlesh_vaghela]

Strange!!

On my instance, it's working .. 😕

Can you please confirm below files structure are available??

[app_name]
bin
[custom_alert_action_script]

default
    alert_actions.conf
    app.conf
    data
        ui
            alerts
                [custom_alert_action].html

README
    alert_actions.conf.spec
    savedsearches.conf.spec

Can we call python script for testing ??

python code (nimsoft.py):

import sys, os, datetime

def log(msg):
f = open(os.path.join(os.environ["SPLUNK_HOME"], "var", "log", "splunk", "test_modalert.log"), "a")
print >> f, str(datetime.datetime.now().isoformat()), msg
f.close()

log("got arguments %s" % sys.argv)
log("got payload: %s" % sys.stdin.read())

print >>sys.stderr, "INFO Hello STDERR"

We are just logging.

log should contain like...

<stanza name="AlertName">
  <param name="result_count">27</param>
  <param name="search_query">search index="_internal" | stats count by sourcetype</param>
</stanza>

Thanks

[rune_hellem]

Folder structure with files

apps\klp_nimsoft_custom_alerts\appserver
apps\klp_nimsoft_custom_alerts\appserver\static
apps\klp_nimsoft_custom_alerts\appserver\static\action.png

apps\klp_nimsoft_custom_alerts\bin
apps\klp_nimsoft_custom_alerts\bin\powershell.path
apps\klp_nimsoft_custom_alerts\bin\README
apps\klp_nimsoft_custom_alerts\bin\scripts
apps\klp_nimsoft_custom_alerts\bin\scripts\testArguments.ps1

apps\klp_nimsoft_custom_alerts\default
apps\klp_nimsoft_custom_alerts\default\alert_actions.conf
apps\klp_nimsoft_custom_alerts\default\app.conf
apps\klp_nimsoft_custom_alerts\default\data
apps\klp_nimsoft_custom_alerts\default\data\ui
apps\klp_nimsoft_custom_alerts\default\data\ui\nav
apps\klp_nimsoft_custom_alerts\default\data\ui\views
apps\klp_nimsoft_custom_alerts\default\data\ui\nav\default.xml
apps\klp_nimsoft_custom_alerts\default\data\ui\views\README

apps\klp_nimsoft_custom_alerts\local
apps\klp_nimsoft_custom_alerts\local\app.conf
apps\klp_nimsoft_custom_alerts\local\savedsearches.conf

apps\klp_nimsoft_custom_alerts\metadata
apps\klp_nimsoft_custom_alerts\metadata\default.meta
apps\klp_nimsoft_custom_alerts\metadata\local.meta

apps\klp_nimsoft_custom_alerts\README
apps\klp_nimsoft_custom_alerts\README\alert_actions.conf.spec

I do not have the savedsearches.conf.spec.

I do not have Python installed on the server so I would prefer to continue using Powershell. When you say it works for you, is that on a Linux install as well? Could it be that the Windows/Powershell combo never has been working?

[kamlesh_vaghela]

Yes, It is Linux.

Let me try Windows/Powershell combo.

[harsmarvania57]

@kamlesh_vaghela & @rune.hellem,

Based on the documentation http://docs.splunk.com/Documentation/Splunk/6.6.0/AdvancedDev/CustomAlertScript#Script_naming_guidel... , stansa which is given in alert_actions.conf and scriptname should be same. However there are no powershell example provided in doc. Based on above configuration it looks like it is different.

I am not big fan of Windows 😛 and I never tried custom alert actions on Windows so I may be wrong but I am running custom alert actions on linux very well. 🙂

[kamlesh_vaghela]

@harsmarvania57

Thanks for your input.

Here we executing the command & It is working properly. As mentioned in question script is executing properly and log output is provided in the question. So here I assumed that script is executing properly. The issue regarding configuration & payload, which is not found.

@rune.hellem,
Kindly correct me if I'm wrong.

[kamlesh_vaghela]

Another idea if you agree.

Can we pass $job.resultCount$ & $job.search$ as an argument of an command??
like.

alert.execute.cmd.arg.4 = $job.resultCount$
alert.execute.cmd.arg.5 =$job.search$

Do we able to access argument 4 & 5 in powerShell??

[rune_hellem]

Added

alert.execute.cmd.arg.4 = $job.resultCount$
alert.execute.cmd.arg.5 =$job.search$

But, unfortunately

[10/27/2017 4:12 PM]: Arg 1: $job.resultCount$
[10/27/2017 4:12 PM]: Arg 2: $job.search$

Does not seem that the variables are accessible in neither scope on Windows other than when logging a generic_single_line

 action.logevent = 1
 action.logevent.param.event = nimsoftCustomAlert=true antall=$result.count$ antallJob=$job.resultCount$
 action.logevent.param.host = splunksearch
 action.logevent.param.index = filenetprod

[rune_hellem]

...and another thing I see is that from at random times, the payload is not being logged either.

[10/27/2017 4:33 PM]: Arg 2: $job.search$
[10/27/2017 4:33 PM]: Settings are: 
[10/27/2017 4:33 PM]: All done

I can try out using a Python script instead, just have to eat the humble pie and install it first. I do not expect that to help me out either, but without giving it a try I won't know...

[kamlesh_vaghela]

Hi
It will be good. but can do the last try??

I found Script override considerations in the below doc. I think due to this

http://docs.splunk.com/Documentation/Splunk/6.6.0/AdvancedDev/CustomAlertScript#Override_a_script_wi...

So can execute power shell from shell script??

1) create powershell-caller.sh in one of the below folder.

$SPLUNK_HOME/bin/scripts
$SPLUNK_HOME/etc/apps/<AppName>/bin/scripts



read sessionKey
powershell.path -NoProfile -f POWER_SHELL_PATH\testArguments.ps1 --execute "'$SPLUNK_ARG_0' '$0' '$1' '$2' '$3' '$4' '$5' '$6' '$7' '$8' '$sessionKey'" 

2) Add below configuration in alert (savsedsearch.conf)

action.script = 1
action.script.filename = powershell-caller.sh

3) Try to access the all arguments passed in the testArguments.ps1 script.

I hope It will work.

Thanks for your co-ordination.

[rune_hellem]

What I've done is to install latest Python on the server. In apps\klp_nimsoft_custom_alerts\bin I have added python.path containing no more than D:\Python\Python37\python.exe

Then in alerts_actions.conf added

[nimsoft_python]
is_custom = 1
label = Nimsoft Python Custom Alert Action
icon_path = action.png
payload_format = json
disabled = 0
alert.execute.cmd = python.path
alert.execute.cmd.arg.0 = $SPLUNK_HOME\etc\apps\klp_nimsoft_custom_alerts\bin\scripts\testArguments.py
param.result_count = $job.resultCount$
param.search_query = $job.search$

The script is (for now) very simple

import sys, os, datetime

def log(msg):
    fileName = open(os.path.join("d:", "temp", "test_modalert.log"), "a")
    msg = str(datetime.datetime.now().isoformat()) + msg 
    print (msg,file=fileName)
    fileName.close()
#endDef

log(" got arguments %s" % sys.argv)

#print ("If executed from command line and not from Spunk, you must break with keyboard")
#log(" got payload: %s" % sys.stdin.read())
#log(sys.stderr)

It does print to the file if executed from command line, but I cannot get it to be triggered by Splunk...and the alert is executed (confirmed by single line event)...thought I had this part under control, same thing I did with Powershell.path...