Why is the custom alert script failing with sendalert?

[jrehl01]

I'm trying to create a custom alert application. All I want to do right now is to see what kind of parameters I can pull and utilize. So I'm just doing a simple print into a file. I created an app called say, test. And the script is called showconfiguration. This is my first time making a custom app.

Here's what the directory structure looks like under /opt/splunk/etc/apps/test:
test/

├── appserver
│   └── static
│       └── icon.png
├── bin
│   └── showconfiguration.py
├── default
│   ├── alert_actions.conf
│   ├── app.conf
│   └── data
│       └── ui
│           └── alerts
│               └── showconfiguration.html
├── metadata
│   ├── default.meta
│   └── local.meta
└── README
    └── alert_actions.conf.spec

Here's alert_actions.conf contents:
> [showconfiguration]
>is_custom = 1
>label = Testing Splunk alerting capability
>description = Testing
>icon_path = icon.png
>payload_format = json
>param.trigger_reason = Saved Search [test] number of events ($job.resultCount$)
>param.result_count = $job.resultCount$
>param.one = two


Contents of app.conf:
> [ui]
>is_visible = 0
>label = Alert Tests
>
>[launcher]
>author = Me
>description = Testing splunk alert capability
>version = 0.1
>
>[install]
>state = enabled
>is_configured = 1

And everything is owned by splunk:splunk and I think has the correct permissions.
So I create an alert and set it to run this custom alert. It never runs and I see this error in the splunkd.log:

08-29-2018 15:10:40.746 -0400 ERROR sendmodalert - Error in 'sendalert' command: Alert action "showconfiguration" not found.

I don't know what I did wrong here. How do I get sendalert to recognize the showconfiguration script??

[jrehl01]

Here is the end result of the script that pretty much shows all the information you can grab about an alert and the corresponding events:

import json, sys, csv, gzip

if __name__ == "__main__":
     if len(sys.argv) > 1 and sys.argv[1] == "--execute":
        data = json.loads(sys.stdin.read())
        f = open("/tmp/splunktest.txt", "w")
        f.write("Here's the info we get from splunk:\n")
        f.write(json.dumps((data), indent=4, sort_keys=False))
        f.write("\n\nResults Data:\n")
        results_file = data["results_file"]

        fz = gzip.open(results_file)
        results_content = csv.DictReader(fz)
        for idx, row in enumerate(results_content):
            f.write("Information for result #" + str(idx) + "\n")
            for key, value in row.iteritems():
                f.write("Key: " + str(key) + "\tValue: " + str(value) + "\n")
            f.write("\n")
        fz.close()
        f.close()

[harsmarvania57 · Accepted solution]

Try below script and do not include #!/usr/bin/python in your script because splunk will use inbuilt python.

import pprint, json, sys

if __name__ == "__main__":
  if len(sys.argv) > 1 and sys.argv[1] = "--execute":
    f.open("/tmp/splunktest.txt", "w")
    f.write("Here's the info we get from splunk:")
    f.write(pprint.pprint(json.loads(sys.stdin.read())))
    f.close()

[pjohnson1]

I'm following this example but get the following error. Any idea of what I did wrong?

04-02-2020 07:30:11.193 +0000 WARN  sendmodalert - action=showconfiguration - Alert action script returned error code=1
04-02-2020 07:30:11.193 +0000 INFO  sendmodalert - action=showconfiguration - Alert action script completed in duration=11 ms with exit code=1
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -  SyntaxError: invalid syntax
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -                                           ^
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -      if len(sys.argv) > 1 and sys.argv[1] = "--execute":
04-02-2020 07:30:11.191 +0000 ERROR sendmodalert - action=showconfiguration STDERR -    File "/opt/splunk/etc/apps/showconfiguration/bin/showconfiguration.py", line 4
host = 6c83f2e55cd4source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

[harsmarvania57]

Please open a new question and refer this question link in new question, also provide your script in new question.

[pjohnson1]

Thanks Harshil.

[jrehl01]

It was more the metadata/default.meta file that was the issue, but I'll still accept you as providing the answer. Thanks again!

[jrehl01]

Unfortunately still didn't help.
Here's the logs again:

08-30-2018 08:05:32.379 -0400 INFO sendmodalert - Invoking modular alert action=showconfiguration for search="Test" sid="rt_scheduler_adminsearch_RMD5742b25d78d6cf18c_at_1535567638_0.185" in app="search" owner="admin" type="saved"

08-30-2018 08:05:32.380 -0400 ERROR sendmodalert - Error in 'sendalert' command: Alert action "showconfiguration" not found.

08-30-2018 08:05:32.380 -0400 ERROR SearchScheduler - Error in 'sendalert' command: Alert action "showconfiguration" not found., search='sendalert showconfiguration results_file="/opt/splunk/var/run/splunk/dispatch/rt_scheduler_adminsearchRMD5742b25d78d6cf18c_at_1535567638_0.185/results.csv.gz" results_link="http://s01-splunk-d01.devmss.leidos.com:8000/app/search/@go?sid=rt_scheduleradminsearch_RMD5742b25d78d6cf18c_at_1535567638_0.185"'

[harsmarvania57]

After changing configuration have you restarted splunk ? If not then please restart splunk.

[jrehl01]

I just restarted and same issue.
However, I'm noticing this piece in the log:

Unable to find alert action script for action="showconfiguration" in app="system"

Why is it looking in the "System" app and not my own? Do I have do update anything in the /opt/splunk/etc/system directory?

[harsmarvania57]

Do you have below configuration in $SPLUNK_HOME/etc/apps/test/metadata/default.meta, if not then please do below settings, restart splunk and try again.

[]
access = read : [ * ], write : [ admin ]

[alert_actions/showconfiguration]
export = system
access = read : [ * ], write : [ admin ]

[alerts]
export = system

[jrehl01]

That was it! I fixed some errors in the script and it successfully ran.
You've been very helpful, thank you!

[harsmarvania57]

Great, I have converted my comment to answer. So that you can accept/upvote it. Also it will be good to provide script which you modified so that other community member will able to use that script in future (Please remove any sensitive data).

[harsmarvania57]

Hi @jrehl01,

showconfiguration.html should be in $SPLUNK_HOME/etc/apps/test/data/ui/alerts/, also it will be good if you will provide content of your python script.

[jrehl01]

It is in that directory. For some reason the spacing that I put in the post isn't reflected when I submitted it.
Here's the content of the script:

#!/usr/bin/python

import pprint, json, sys

if name == "main":
f.open("/tmp/splunktest.txt", "w")
f.write("Here's the info we get from splunk:")
f.write(pprint.pprint(json.loads(sys.stdin.read())))
f.close()

[jrehl01]

Oh I didn't even notice the formatting was off on the tree. It is under that directory, the formatting of this post just doesn't reflect that. Not sure why I can't update the amount of spacing there.

Here's the content of the script:

!/usr/bin/python

import pprint, json, sys

if __name__ == "__main__":
f.open("/tmp/splunktest.txt", "w")
f.write("Here's the info we get from splunk:")
f.write(pprint.pprint(json.loads(sys.stdin.read())))
f.close()