topic Re: Can't get value of job.resultCount when using Custom Alert Action in Getting Data In

Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Wed, 25 Oct 2017 06:10:52 GMT

How the alert is defined

I have created a custom alert action after following documentation found here http://docs.splunk.com/Documentation/Splunk/6.6.0/AdvancedDev/ModAlertsIntro, my alert is defined like this

[nimsoft]
is_custom = 1
label = Nimsoft Custom Alert Action
icon_path = action.png
payload_format = json
disabled = 0
alert.execute.cmd = powershell.path
alert.execute.cmd.arg.0 = -NoProfile
alert.execute.cmd.arg.1 = -f
alert.execute.cmd.arg.2 = $SPLUNK_HOME\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1
alert.execute.cmd.arg.3 = --execute
param.result_count = $job.resultCount$
param.search_query = $job.search$

Problem description

The above alert is working almost just fine. Using Powershell I am able to get hold of both the payload and the command line arguments, script output is like this

[10/19/2017 8:23 AM]: Now loop all arguments
[10/19/2017 8:23 AM]: Arg 0: --execute
[10/19/2017 8:23 AM]: Settings are: @{app=klp_nimsoft_custom_alerts; owner=admin; results_file=D:\splunk\var\run\splunk\dispatch\scheduler__admin_...__TestAlarm_at_1508394180_23089\per_result_alert\tmp_24.csv.gz; results_link=http://SplunkSearch:80/app/klp_nimsoft_custom_alerts/search?q=%7Cloadjob%20scheduler__admin_....w__TestAlarm_at_....&earliest=0&latest=now; search_uri=/servicesNS/nobody/klp_nimsoft_custom_alerts/saved/searches/TestAlarm; server_host=SPLUNKSEARCH; server_uri=https://127.0.0.1:8089; session_key=iwb0t_....; sid=scheduler__admin_...__TestAlarm_at_1508394180_23089; search_name=TestAlarm; configuration=; result=}
[10/19/2017 8:23 AM]: All done

But problem is that in order to really do the magic I need to to with the proper script, I need to know the number of events found by the alert, that is

  param.result_count = $job.resultCount$

But the value is nowhere to be found, at least it can be found where I expect it to be found, so maybe I need to look into other places. Anyone able to see what's wrong here?

The Powershell script testArguments.ps1

It might be that someone is curious about how the script looks like as well, so here goes

<#
.Synopsis
Script used to verify Splunk alerts, write to a log file both command line argumenst and stdin (payload)

.Description
Powershell -File "D:\Splunk\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1"

#>


<#
    Get current timestamp, used when writing to logfile
#>
function Get-TimeStamp {

    $timeStamp = "[" + (Get-Date).ToShortDateString() + " " + ((Get-Date).ToShortTimeString()) + "]"

    Return $timeStamp

}

# The logfile
$fileName = "d:\temp\arguments_test_updated.log"

# If exist, remove
If (Test-Path $fileName) {
    Remove-Item $fileName
}

# Start printing all argv's
$msg = (Get-TimeStamp) + ": Now loop all arguments"
write-host  $msg
Add-Content $fileName $msg
for ( $i = 0; $i -lt $args.count; $i++ ) {
    $msg = (Get-TimeStamp) + ": Arg $($i): $($args[$i])"
    write-host $msg
    Add-Content $fileName $msg

}

# Print stdin
# https://stackoverflow.com/questions/44695956/what-is-powershells-equivalent-to-pythons-sys-stdin-read
$settings = $input | Out-String | ConvertFrom-Json
$msg = (Get-TimeStamp) + ": Settings are: " + $settings
write-host $msg
Add-Content $fileName $msg
$msg = (Get-TimeStamp) +  ": All done"
Write-Host $msg
Add-Content $fileName $msg

Re: Can't get value of job.resultCount when using Custom Alert Action

harsmarvania57 — Wed, 25 Oct 2017 09:58:24 GMT

Hi @rune.hellem,

It looks like you are not getting param.result_count and param.search_query result in your payload, those value should come after configuration=.

You can do one thing try to create spec file in $SPLUNK_HOME/etc/apps/<CUSTOM ALERT APP>/README/alert_actions.conf.spec with below content

[nimsoft]

param.result_count = <integer>
* Runtime value of result count.

param.search_query = <string>
* Runtime splunk search query.

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Thu, 26 Oct 2017 07:56:34 GMT

I did create file D:\splunk\etc\apps\klp_nimsoft_custom_alerts\README\alert_actions.conf.spec with content

[nimsoft]
# Runtime value of result count. 
param.result_count = <integer>

# Runtime splunk search query. 
param.search_query = <string>

Then first attempt, just did a refresh http://.../debug/refresh, did not help. Then restarted, still same, output is still

...search_name=TestAlarm; configuration=; result=}

Re: Can't get value of job.resultCount when using Custom Alert Action

harsmarvania57 — Thu, 26 Oct 2017 08:08:22 GMT

What is your splunk query and when you run it manually, are you getting any results ? Based on your payload results.csv.gz file is generating but you are not getting any value after result=, that is quite strange.

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Tue, 29 Sep 2020 16:28:55 GMT

Have tried to pass hardcode value in conf file??

like..
param.result_count = 100
param.search_query = | stats count

Does payload populate as expected?
Thanks

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Thu, 26 Oct 2017 09:14:32 GMT

index=*prod error

It would have been nice to be able to say that the above search string does not return results...but...it does 🙂

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Thu, 26 Oct 2017 09:39:02 GMT

Did just now try

param.result_count = 67
param.search_query = | stats count

First /debug/refresh, then restart. Still

 search_name=TestAlarm; configuration=; result=}

To be all sure, did search for result_count in all files in folder

D:\splunk\var\run\splunk\dispatch\scheduler__admin_a2xwX25pbXNvZnRfY3VzdG9tX2FsZXJ0cw__TestAlarm_at_1509009960_21

Found nothing. Did also check the results.csv.gz and it does have results.

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Thu, 26 Oct 2017 09:53:34 GMT

can you share your savedsearch.conf configuration for this particular alert?

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Tue, 29 Sep 2020 16:29:06 GMT

[TestAlarm]
action.email.useNSSubject = 1
action.logevent = 1
action.logevent.param.event = nimsoftCustomAlert=true antall=$result.count$ antallJob=$job.resultCount$
action.logevent.param.host = splunksearch
action.logevent.param.index = filenetprod
action.nimsoft = 1
action.nimsoft_100_filenet_error = 1
alert.digest_mode = 0
alert.suppress = 0
alert.track = 0
counttype = number of events
cron_schedule = * * * * *
dispatch.earliest_time = -2m
dispatch.latest_time = now
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = klp_nimsoft_custom_alerts
request.ui_dispatch_view = search
search = index=filenetprod error

The action logevent creates an event of type generic_single_line which is logged like this
nimsoftCustomAlert=true antall= antallJob=44

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Thu, 26 Oct 2017 12:31:02 GMT

can please add below two entry in [TestAlarm]

action.nimsoft.param.result_count = $job.resultCount$
action.nimsoft.param.search_query = $job.search$

Meanwhile can you please check any error during executing of search?

index=_internal component=SavedSplunker savedsearch_name="TestAlarm"

index=_internal component=script command=runshellscript

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Thu, 26 Oct 2017 12:56:42 GMT

Hmm, hard nail this one...

Did add to savesearches.conf

action.nimsoft.param.result_count = $job.resultCount$
action.nimsoft.param.search_query = $job.search$

Restarted Splunk, still same

...search_name=TestAlarm; configuration=; result=}

Checking internal log, only info messages

10-26-2017 14:49:29.685 +0200 INFO  SavedSplunker - savedsearch_id="nobody;klp_nimsoft_custom_alerts;TestAlarm", search_type="scheduled", user="admin", app="klp_nimsoft_custom_alerts", savedsearch_name="TestAlarm", priority=default, status=success, digest_mode=0, scheduled_time=1509022140, window_time=0, dispatch_time=1509022141, run_time=0.875, result_count=23, alert_actions="logevent,nimsoft", sid="scheduler__admin_a2xwX25pbXNvZnRfY3VzdG9tX2FsZXJ0cw__TestAlarm_at_1509022140_14", suppressed=0, fired=23, skipped=0, action_time_ms=26641, thread_id="AlertNotifierWorker-0", message=""

and

index=_internal component=ScriptRunner

actually does return

10-26-2017 14:45:54.463 +0200 ERROR ScriptRunner - Couldn't start child process. script="C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -f D:\splunk\etc\apps\klp_nimsoft_custom_alerts\bin\testArguments.ps1 --execute"

But it seems to be a false positive, since the log file is being updated.

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Thu, 26 Oct 2017 13:10:51 GMT

mmm,

Is below is another alert action or something in [TestAlarm] ?? Can we comment It?
like..

#action.nimsoft_100_filenet_error = 1

Apart from it, can we validate configuration by executing btool command??

YOUR_SPLUNK_PATH/bin/splunk btool check

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Thu, 26 Oct 2017 13:58:43 GMT

Did clean up the alert, now it is defined like this

[TestAlarm]
action.email.useNSSubject = 1
action.nimsoft = 1
action.nimsoft.param.result_count = $job.resultCount$
action.nimsoft.param.search_query = $job.search$
alert.digest_mode = 0
alert.suppress = 0
alert.track = 0
counttype = number of events
cron_schedule = * * * * *
dispatch.earliest_time = -2m
dispatch.latest_time = now
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_app = klp_nimsoft_custom_alerts
request.ui_dispatch_view = search
search = index=filenetprod error

Restarted the server to be all sure, but still no help. Nothing in configuration yet.
Splunk btool check
does not show any issuew with the app.

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Tue, 29 Sep 2020 16:30:03 GMT

Strange!!

On my instance, it's working .. 😕

Can you please confirm below files structure are available??

[app_name]
bin
[custom_alert_action_script]

default
    alert_actions.conf
    app.conf
    data
        ui
            alerts
                [custom_alert_action].html

README
    alert_actions.conf.spec
    savedsearches.conf.spec

Can we call python script for testing ??

python code (nimsoft.py):

import sys, os, datetime

def log(msg):
f = open(os.path.join(os.environ["SPLUNK_HOME"], "var", "log", "splunk", "test_modalert.log"), "a")
print >> f, str(datetime.datetime.now().isoformat()), msg
f.close()

log("got arguments %s" % sys.argv)
log("got payload: %s" % sys.stdin.read())

print >>sys.stderr, "INFO Hello STDERR"

We are just logging.

log should contain like...

<stanza name="AlertName">
  <param name="result_count">27</param>
  <param name="search_query">search index="_internal" | stats count by sourcetype</param>
</stanza>

Thanks

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Fri, 27 Oct 2017 11:16:58 GMT

Folder structure with files

apps\klp_nimsoft_custom_alerts\appserver
apps\klp_nimsoft_custom_alerts\appserver\static
apps\klp_nimsoft_custom_alerts\appserver\static\action.png

apps\klp_nimsoft_custom_alerts\bin
apps\klp_nimsoft_custom_alerts\bin\powershell.path
apps\klp_nimsoft_custom_alerts\bin\README
apps\klp_nimsoft_custom_alerts\bin\scripts
apps\klp_nimsoft_custom_alerts\bin\scripts\testArguments.ps1

apps\klp_nimsoft_custom_alerts\default
apps\klp_nimsoft_custom_alerts\default\alert_actions.conf
apps\klp_nimsoft_custom_alerts\default\app.conf
apps\klp_nimsoft_custom_alerts\default\data
apps\klp_nimsoft_custom_alerts\default\data\ui
apps\klp_nimsoft_custom_alerts\default\data\ui\nav
apps\klp_nimsoft_custom_alerts\default\data\ui\views
apps\klp_nimsoft_custom_alerts\default\data\ui\nav\default.xml
apps\klp_nimsoft_custom_alerts\default\data\ui\views\README

apps\klp_nimsoft_custom_alerts\local
apps\klp_nimsoft_custom_alerts\local\app.conf
apps\klp_nimsoft_custom_alerts\local\savedsearches.conf

apps\klp_nimsoft_custom_alerts\metadata
apps\klp_nimsoft_custom_alerts\metadata\default.meta
apps\klp_nimsoft_custom_alerts\metadata\local.meta

apps\klp_nimsoft_custom_alerts\README
apps\klp_nimsoft_custom_alerts\README\alert_actions.conf.spec

I do not have the savedsearches.conf.spec.

I do not have Python installed on the server so I would prefer to continue using Powershell. When you say it works for you, is that on a Linux install as well? Could it be that the Windows/Powershell combo never has been working?

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Fri, 27 Oct 2017 11:22:15 GMT

Yes, It is Linux.

Let me try Windows/Powershell combo.

Re: Can't get value of job.resultCount when using Custom Alert Action

harsmarvania57 — Fri, 27 Oct 2017 11:29:07 GMT

@kamlesh_vaghela & @rune.hellem,

Based on the documentation http://docs.splunk.com/Documentation/Splunk/6.6.0/AdvancedDev/CustomAlertScript#Script_naming_guidelines , stansa which is given in alert_actions.conf and scriptname should be same. However there are no powershell example provided in doc. Based on above configuration it looks like it is different.

I am not big fan of Windows 😛 and I never tried custom alert actions on Windows so I may be wrong but I am running custom alert actions on linux very well. 🙂

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Fri, 27 Oct 2017 11:47:54 GMT

@harsmarvania57

Thanks for your input.

Here we executing the command & It is working properly. As mentioned in question script is executing properly and log output is provided in the question. So here I assumed that script is executing properly. The issue regarding configuration & payload, which is not found.

@rune.hellem,
Kindly correct me if I'm wrong.

Re: Can't get value of job.resultCount when using Custom Alert Action

kamlesh_vaghela — Fri, 27 Oct 2017 11:49:32 GMT

Another idea if you agree.

Can we pass $job.resultCount$ & $job.search$ as an argument of an command??
like.

alert.execute.cmd.arg.4 = $job.resultCount$
alert.execute.cmd.arg.5 =$job.search$

Do we able to access argument 4 & 5 in powerShell??

Re: Can't get value of job.resultCount when using Custom Alert Action

rune_hellem — Tue, 29 Sep 2020 16:30:21 GMT

Added

alert.execute.cmd.arg.4 = $job.resultCount$
alert.execute.cmd.arg.5 =$job.search$

But, unfortunately

[10/27/2017 4:12 PM]: Arg 1: $job.resultCount$
[10/27/2017 4:12 PM]: Arg 2: $job.search$

Does not seem that the variables are accessible in neither scope on Windows other than when logging a generic_single_line

 action.logevent = 1
 action.logevent.param.event = nimsoftCustomAlert=true antall=$result.count$ antallJob=$job.resultCount$
 action.logevent.param.host = splunksearch
 action.logevent.param.index = filenetprod