Can't get value of job.resultCount when using Custom Alert Action

[rune_hellem]

How the alert is defined

I have created a custom alert action after following documentation found here http://docs.splunk.com/Documentation/Splunk/6.6.0/AdvancedDev/ModAlertsIntro, my alert is defined like this

[nimsoft]
is_custom = 1
label = Nimsoft Custom Alert Action
icon_path = action.png
payload_format = json
disabled = 0
alert.execute.cmd = powershell.path
alert.execute.cmd.arg.0 = -NoProfile
alert.execute.cmd.arg.1 = -f
alert.execute.cmd.arg.2 = $SPLUNK_HOME\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1
alert.execute.cmd.arg.3 = --execute
param.result_count = $job.resultCount$
param.search_query = $job.search$

Problem description

The above alert is working almost just fine. Using Powershell I am able to get hold of both the payload and the command line arguments, script output is like this

[10/19/2017 8:23 AM]: Now loop all arguments
[10/19/2017 8:23 AM]: Arg 0: --execute
[10/19/2017 8:23 AM]: Settings are: @{app=klp_nimsoft_custom_alerts; owner=admin; results_file=D:\splunk\var\run\splunk\dispatch\scheduler__admin_...__TestAlarm_at_1508394180_23089\per_result_alert\tmp_24.csv.gz; results_link=http://SplunkSearch:80/app/klp_nimsoft_custom_alerts/search?q=%7Cloadjob%20scheduler__admin_....w__TestAlarm_at_....&earliest=0&latest=now; search_uri=/servicesNS/nobody/klp_nimsoft_custom_alerts/saved/searches/TestAlarm; server_host=SPLUNKSEARCH; server_uri=https://127.0.0.1:8089; session_key=iwb0t_....; sid=scheduler__admin_...__TestAlarm_at_1508394180_23089; search_name=TestAlarm; configuration=; result=}
[10/19/2017 8:23 AM]: All done

But problem is that in order to really do the magic I need to to with the proper script, I need to know the number of events found by the alert, that is

  param.result_count = $job.resultCount$

But the value is nowhere to be found, at least it can be found where I expect it to be found, so maybe I need to look into other places. Anyone able to see what's wrong here?

The Powershell script testArguments.ps1

It might be that someone is curious about how the script looks like as well, so here goes

<#
.Synopsis
Script used to verify Splunk alerts, write to a log file both command line argumenst and stdin (payload)

.Description
Powershell -File "D:\Splunk\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1"

#>


<#
    Get current timestamp, used when writing to logfile
#>
function Get-TimeStamp {

    $timeStamp = "[" + (Get-Date).ToShortDateString() + " " + ((Get-Date).ToShortTimeString()) + "]"

    Return $timeStamp

}

# The logfile
$fileName = "d:\temp\arguments_test_updated.log"

# If exist, remove
If (Test-Path $fileName) {
    Remove-Item $fileName
}

# Start printing all argv's
$msg = (Get-TimeStamp) + ": Now loop all arguments"
write-host  $msg
Add-Content $fileName $msg
for ( $i = 0; $i -lt $args.count; $i++ ) {
    $msg = (Get-TimeStamp) + ": Arg $($i): $($args[$i])"
    write-host $msg
    Add-Content $fileName $msg

}

# Print stdin
# https://stackoverflow.com/questions/44695956/what-is-powershells-equivalent-to-pythons-sys-stdin-rea...
$settings = $input | Out-String | ConvertFrom-Json
$msg = (Get-TimeStamp) + ": Settings are: " + $settings
write-host $msg
Add-Content $fileName $msg
$msg = (Get-TimeStamp) +  ": All done"
Write-Host $msg
Add-Content $fileName $msg

[kamlesh_vaghela]

Hi @rune.hellem,
Apology for delayed.

I did few changes in your code.

In alerts_actions.conf,

[nimsoft_python]
is_custom = 1
label = Nimsoft Python Custom Alert Action
icon_path = action.png
payload_format = json
disabled = 0
param.result_count = $job.resultCount$
param.search_query = $job.search$

// removed command line code and added new file nimsoft_python.py.

create nimsoft_python.py file in bin folder. (keep file name nimsoft_python.py only.)

import sys, os, datetime

def log(msg):
  fileName = open(os.path.join("d:", "temp", "test_modalert.log"), "a")
  msg = str(datetime.datetime.now().isoformat()) + msg 
  print (msg,file=fileName)
  fileName.close()
#endDef

log(" got payload: %s" % sys.stdin.read())

Splunk alert action should execute this file.

Can you please try it?

Thanks

[rune_hellem]

Took a while, sorry for that, but I did update the alert as described, but ... doh, no help.

Have verified that the script can be called from command line, just a minor change and then it worked

fileName = open(os.path.join("d:/", "temp", "test_modalert.log"), "a")

d:/, then I could run the script by just typing nimsoft_python.py and it would log from the bin-folder. But still not triggered by the alert. Tried to move the file to bin\scripts, did always restart Splunk, but no. Still won't work .... tough nut to crack this one

[kamlesh_vaghela]

Hi,
Strange!! We looked every corner of code.

Is it possible to share your code (use sample values in-do not change filename) ? So I trace it in my lab?
2nd, how you are checking/troubleshooting alerts?

Thanks

[rune_hellem]

1. Should I create a support ticket and upload a diag-file?
2. I check the alerts using generic_single_line action.logevent = 1 action.logevent.param.event = alarm=$name$ antall=$job.resultCount$ Rune action.logevent.param.host = splunksearch action.logevent.param.index = filenetprod

[kamlesh_vaghela]

Yeah, support ticket will help us to trace it. They will find where we missed.
I'm waiting for that missed point.

Thanks for you long coordination.

[rune_hellem]

File Received
Thanks for your file upload. Your request has been sent to support.

So, by referring to what is described here and the diag file, hopefully that will help. Crossing fingers for a complex fix required and not a simple typo that we have missed 🙂

[kamlesh_vaghela]

Hi @rune.hellem,

Any Update?

[rune_hellem]

Now working with Support. First lesson learned that I did discover by chance right now is that I did not need to install Python for Windows since Splunk uses its own Python.exe, which is pre Python 3.7 which I did install.

From the _internal-index I found 11-27-2017 10:35:05.767 +0100 ERROR sendmodalert - action=nimsoft_python STDERR - File "D:\splunk\etc\apps\klp_nimsoft_custom_alerts\bin\nimsoft_python.py", line 22 caused by print (msg, file=$filename. Splunk Python wants it the old way print >> f, str(datetime.datetime.now().isoformat()), msg

And when I got that working I found

"result_count":"13"

So that was how to get Python working. Still not solved Powershell, working on that now

[rune_hellem]

Did create the support ticket at the end of last week. Confirmed recieved and investigation started, waiting for feedback.

[harsmarvania57]

Hi @rune.hellem,

It looks like you are not getting param.result_count and param.search_query result in your payload, those value should come after configuration=.

You can do one thing try to create spec file in $SPLUNK_HOME/etc/apps/<CUSTOM ALERT APP>/README/alert_actions.conf.spec with below content

[nimsoft]

param.result_count = <integer>
* Runtime value of result count.

param.search_query = <string>
* Runtime splunk search query.

[rune_hellem]

I did create file D:\splunk\etc\apps\klp_nimsoft_custom_alerts\README\alert_actions.conf.spec with content

[nimsoft]
# Runtime value of result count. 
param.result_count = <integer>

# Runtime splunk search query. 
param.search_query = <string>

Then first attempt, just did a refresh http://.../debug/refresh, did not help. Then restarted, still same, output is still

...search_name=TestAlarm; configuration=; result=}

[harsmarvania57]

What is your splunk query and when you run it manually, are you getting any results ? Based on your payload results.csv.gz file is generating but you are not getting any value after result=, that is quite strange.

[rune_hellem]

index=*prod error

It would have been nice to be able to say that the above search string does not return results...but...it does 🙂