topic Why is the custom alert script failing with sendalert? in Alerting

Why is the custom alert script failing with sendalert?

jrehl01 — Tue, 09 Jun 2020 22:02:19 GMT

I'm trying to create a custom alert application. All I want to do right now is to see what kind of parameters I can pull and utilize. So I'm just doing a simple print into a file. I created an app called say, test. And the script is called showconfiguration. This is my first time making a custom app.

Here's what the directory structure looks like under /opt/splunk/etc/apps/test:
test/

├── appserver
│   └── static
│       └── icon.png
├── bin
│   └── showconfiguration.py
├── default
│   ├── alert_actions.conf
│   ├── app.conf
│   └── data
│       └── ui
│           └── alerts
│               └── showconfiguration.html
├── metadata
│   ├── default.meta
│   └── local.meta
└── README
    └── alert_actions.conf.spec

Here's alert_actions.conf contents:
> [showconfiguration]
>is_custom = 1
>label = Testing Splunk alerting capability
>description = Testing
>icon_path = icon.png
>payload_format = json
>param.trigger_reason = Saved Search [test] number of events ($job.resultCount$)
>param.result_count = $job.resultCount$
>param.one = two


Contents of app.conf:
> [ui]
>is_visible = 0
>label = Alert Tests
>
>[launcher]
>author = Me
>description = Testing splunk alert capability
>version = 0.1
>
>[install]
>state = enabled
>is_configured = 1

And everything is owned by splunk:splunk and I think has the correct permissions.
So I create an alert and set it to run this custom alert. It never runs and I see this error in the splunkd.log:

08-29-2018 15:10:40.746 -0400 ERROR sendmodalert - Error in 'sendalert' command: Alert action "showconfiguration" not found.

I don't know what I did wrong here. How do I get sendalert to recognize the showconfiguration script??

Re: Why is the custom alert script failing with sendalert?

harsmarvania57 — Thu, 30 Aug 2018 08:39:34 GMT

Hi @jrehl01,

showconfiguration.html should be in $SPLUNK_HOME/etc/apps/test/data/ui/alerts/, also it will be good if you will provide content of your python script.

Re: Why is the custom alert script failing with sendalert?

jrehl01 — Thu, 30 Aug 2018 11:46:19 GMT

Oh I didn't even notice the formatting was off on the tree. It is under that directory, the formatting of this post just doesn't reflect that. Not sure why I can't update the amount of spacing there.

Here's the content of the script:

!/usr/bin/python

import pprint, json, sys

if __name__ == "__main__":
f.open("/tmp/splunktest.txt", "w")
f.write("Here's the info we get from splunk:")
f.write(pprint.pprint(json.loads(sys.stdin.read())))
f.close()

Re: Why is the custom alert script failing with sendalert?

jrehl01 — Thu, 30 Aug 2018 11:48:19 GMT

It is in that directory. For some reason the spacing that I put in the post isn't reflected when I submitted it.
Here's the content of the script:

#!/usr/bin/python

import pprint, json, sys

if name == "main":
f.open("/tmp/splunktest.txt", "w")
f.write("Here's the info we get from splunk:")
f.write(pprint.pprint(json.loads(sys.stdin.read())))
f.close()

Re: Why is the custom alert script failing with sendalert?

harsmarvania57 — Thu, 30 Aug 2018 11:59:56 GMT

Try below script and do not include #!/usr/bin/python in your script because splunk will use inbuilt python.

import pprint, json, sys

if __name__ == "__main__":
  if len(sys.argv) > 1 and sys.argv[1] = "--execute":
    f.open("/tmp/splunktest.txt", "w")
    f.write("Here's the info we get from splunk:")
    f.write(pprint.pprint(json.loads(sys.stdin.read())))
    f.close()

Re: Why is the custom alert script failing with sendalert?

jrehl01 — Tue, 29 Sep 2020 21:05:29 GMT

Unfortunately still didn't help.
Here's the logs again:

08-30-2018 08:05:32.379 -0400 INFO sendmodalert - Invoking modular alert action=showconfiguration for search="Test" sid="rt_scheduler_adminsearch_RMD5742b25d78d6cf18c_at_1535567638_0.185" in app="search" owner="admin" type="saved"

08-30-2018 08:05:32.380 -0400 ERROR sendmodalert - Error in 'sendalert' command: Alert action "showconfiguration" not found.

08-30-2018 08:05:32.380 -0400 ERROR SearchScheduler - Error in 'sendalert' command: Alert action "showconfiguration" not found., search='sendalert showconfiguration results_file="/opt/splunk/var/run/splunk/dispatch/rt_scheduler_adminsearchRMD5742b25d78d6cf18c_at_1535567638_0.185/results.csv.gz" results_link="http://s01-splunk-d01.devmss.leidos.com:8000/app/search/@go?sid=rt_scheduleradminsearch_RMD5742b25d78d6cf18c_at_1535567638_0.185"'

Re: Why is the custom alert script failing with sendalert?

harsmarvania57 — Thu, 30 Aug 2018 12:14:50 GMT

After changing configuration have you restarted splunk ? If not then please restart splunk.

Re: Why is the custom alert script failing with sendalert?

jrehl01 — Thu, 30 Aug 2018 12:39:11 GMT

I just restarted and same issue.
However, I'm noticing this piece in the log:

Unable to find alert action script for action="showconfiguration" in app="system"

Why is it looking in the "System" app and not my own? Do I have do update anything in the /opt/splunk/etc/system directory?

Re: Why is the custom alert script failing with sendalert?

harsmarvania57 — Thu, 30 Aug 2018 12:56:24 GMT

Do you have below configuration in $SPLUNK_HOME/etc/apps/test/metadata/default.meta, if not then please do below settings, restart splunk and try again.

[]
access = read : [ * ], write : [ admin ]

[alert_actions/showconfiguration]
export = system
access = read : [ * ], write : [ admin ]

[alerts]
export = system

Re: Why is the custom alert script failing with sendalert?

jrehl01 — Thu, 30 Aug 2018 13:08:37 GMT

That was it! I fixed some errors in the script and it successfully ran.
You've been very helpful, thank you!

Re: Why is the custom alert script failing with sendalert?

harsmarvania57 — Thu, 30 Aug 2018 13:22:21 GMT

Great, I have converted my comment to answer. So that you can accept/upvote it. Also it will be good to provide script which you modified so that other community member will able to use that script in future (Please remove any sensitive data).

Re: Why is the custom alert script failing with sendalert?

jrehl01 — Thu, 30 Aug 2018 13:27:46 GMT

It was more the metadata/default.meta file that was the issue, but I'll still accept you as providing the answer. Thanks again!

Re: Why is the custom alert script failing with sendalert?

jrehl01 — Thu, 30 Aug 2018 14:53:26 GMT

Here is the end result of the script that pretty much shows all the information you can grab about an alert and the corresponding events:

import json, sys, csv, gzip

if __name__ == "__main__":
     if len(sys.argv) > 1 and sys.argv[1] == "--execute":
        data = json.loads(sys.stdin.read())
        f = open("/tmp/splunktest.txt", "w")
        f.write("Here's the info we get from splunk:\n")
        f.write(json.dumps((data), indent=4, sort_keys=False))
        f.write("\n\nResults Data:\n")
        results_file = data["results_file"]

        fz = gzip.open(results_file)
        results_content = csv.DictReader(fz)
        for idx, row in enumerate(results_content):
            f.write("Information for result #" + str(idx) + "\n")
            for key, value in row.iteritems():
                f.write("Key: " + str(key) + "\tValue: " + str(value) + "\n")
            f.write("\n")
        fz.close()
        f.close()

Re: Why is the custom alert script failing with sendalert?

pjohnson1 — Thu, 02 Apr 2020 07:42:03 GMT

I'm following this example but get the following error. Any idea of what I did wrong?

04-02-2020 07:30:11.193 +0000 WARN  sendmodalert - action=showconfiguration - Alert action script returned error code=1
04-02-2020 07:30:11.193 +0000 INFO  sendmodalert - action=showconfiguration - Alert action script completed in duration=11 ms with exit code=1
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -  SyntaxError: invalid syntax
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -                                           ^
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -      if len(sys.argv) > 1 and sys.argv[1] = "--execute":
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -    File "/opt/splunk/etc/apps/showconfiguration/bin/showconfiguration.py", line 4
host = 6c83f2e55cd4source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

Re: Why is the custom alert script failing with sendalert?

harsmarvania57 — Thu, 02 Apr 2020 10:40:04 GMT

Please open a new question and refer this question link in new question, also provide your script in new question.

Re: Why is the custom alert script failing with sendalert?

pjohnson1 — Fri, 03 Apr 2020 04:49:50 GMT

Thanks Harshil.