Jira alert not. triggering

[dpark1113]

Hi,

I have Jira 1.0.7 installed on Splunk Cloud. I have 2 actions setup: Jira and Email.

When I ran a test yesterday, both alerts triggered. Today only triggered the Email. The user is already in jira_alert_action group.

There is no errors or creation in Jira dashboard. Any idea what I need to do here? Is there a grace period when the next Jira alert gets trigger?

Thanks!

[guilmxm]

Hi @dpark1113

Every alert triggering that would result in the jira alert action being called would mean traces in the logs, have you checked the embedded report ? (in case something went wrong and wasn't catch by the dashboard)

JIRA Service Desk - modular action logs

If the action was called there will be traces, a common misconception would be the throttling options of alerts, which will / will not depending on your settings trigger the action.
If this is a correlation search in ES, you need as well to make sure the correlation search triggers for each results by editing the alert in the search editor (Settings / Alerts and reports) rather than the correlation search editor.

Check the logs and revert if required.

Guilhem

[dpark1113]

First, thanks for your response.

I don't think I explained correctly in my first post. I have 2 actions in a single alert, only one action gets trigger which is email.

There are no any logs in "modular action logs" for today, I take no Jira alert was triggered?

I'm not sure what "correlation search in ES" means but here is the alert setting.

There is only one result from search.

Blockquote

Alert Type: Scheduled
Run on Cron Schedule

Time Range: Last 30 seconds

Cron Expression: * / 5 * * * *

Expires: 24 hours

Trigger alert when: Number of Results
is equal to 0

Trigger: Once
Throttle: Checked
Suppress triggering for: 60 minutes

[guilmxm]

No problem.

Basically, When Splunk triggers an alert action (a correlation search a specific type of alert for Enterprise Security), the first activity is logged in splunkd, in the case of JIRA:

index=_internal source=*splunkd.log sendmodalert action=jira_service_desk*

Other type of action such as send an email is an alert action as well, and there would be traces in splunkd too.

Then when Splunk starts the action, you will start to get logs in the TA logs as Splunk passes the action to it.

if you start first by splunkd logs, you would find an evidence if Splunk really ran the acton or not.

Then I would say certainly your alert configuration does not make much sense, the time range is too short compared to the cron expression, you could look at the last 10 min every 5 min if you need some extra margin in case of late data arrival, then having your throttling option if you need it depending on your case.

The fact that trigger once, means the alert will never trigger more than once in 60 minutes no matters the number of results, you can as well have throttling options, trigger for each and have a by statement key (like a stats by host or by user) which means Splunk triggers the action for each by statement object, in respect with your settings

If there are any error triggering the action, lack a lack of privileges, this will only be visible in splunkd log because the action will not reach the TA backend.

Guilhem

[guilmxm]

I have seen a message talking a the following reply, but can't seem to see it online @dpark1113 ?

"I ran the search and I see lot of error messages.

I don't know how this was working yesterday and before. Maybe something got changed in Splunk Cloud."

Which errors ?

[dpark1113]

I posted the errors here and moderator is reviewing the post. I guess it'll show up later.

[dpark1113]

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - UnboundLocalError: local variable 'rf' referenced before assignment
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - if rf:
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/alert_actions_base.py", line 204, in prepare_meta_for_cam
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - self.prepare_meta_for_cam()
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/alert_actions_base.py", line 215, in run
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - exitcode = AlertActionWorkerjira_service_desk("TA-jira-service-desk-simple-addon", "jira_service_desk").run(sys.argv)
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/jira_service_desk.py", line 72, in
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - Traceback (most recent call last):
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - During handling of the above exception, another exception occurred:
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - FileNotFoundError: [Errno 2] No such file or directory: '/opt/splunk/var/run/splunk/dispatch/scheduler_ZWdyYW50QGFzc2V0LWNvbnRyb2wuY29t_VEEtamlyYS1zZXJ2aWNlLWRlc2stc2ltcGxlLWFkZG9u_RMD57a6fb1e9edd3d4b9_at_1590005700_14565/results.srs.zst'
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/lib/python3.7/gzip.py", line 163, in __init_
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - binary_file = GzipFile(filename, gz_mode, compresslevel)
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/lib/python3.7/gzip.py", line 53, in open
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

[guilmxm]

@dpark1113

These isues aren't specific to the TA, I think the search head encountered some failures, either splunkd was in a bad state or it was re-created by the Cloud somehow.

I would advise you to perform new tests, you can easily create an alert that generates a results and activate the JIRA action.

| makeresults
| eval user="dummy", action="failure", jira_issue_message="Some message you recycle in JIRA"

[dpark1113]

I created an alert with 2 actions based on that search string. It works and creating incident tickets so far. First one and another one after 60 suppressed minutes.

Similar alert with different search string only triggering email action only. Any idea why that is?

Thanks!

[guilmxm]

@dpark1113

I guess you meant the alert that triggers an email but no the JIRA action earlier.
Honestly without having my hands on the keyboard it's a bit hard to say, all I can say is the addon works fine and if you didn't have logs in splunkd showing a failure while attempting to run the action, or logs from the action itself showing a failure, the JIRA addon is definitively not in cause 😉

[dpark1113]

Hi @guilmxm,

I opened a ticket with Splunk and they suggest to restarting the server so they did. Still having the issue. I deleted the alert and created again, no go.

So I start looking at the logs and trace them one by one.

So here they are.

1. alert_actions_base.py", line 195
2. gzip.open(self.results_file, 'rt')
3. /opt/splunk/lib/python3.7/gzip.py", line 53, in open
4. binary_file = GzipFile(filename, gz_mode, compresslevel)
5. "/opt/splunk/lib/python3.7/gzip.py", line 163, in init
6. fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
7. Errno 2] No such file or directory: '/opt/splunk/var/run/splunk/dispatch/scheduler_ZWdyYW50QGFzc2V0LWNvbnRyb2wuY29t_VEEtamlyYS1zZXJ2aWNlLWRlc2stc2ltcGxlLWFkZG9u__RMD53859ccd8dd824bba_at_1590179400_936/results.srs.zst'

So if I'm going to guess, the file results.srs.zst doesn't exist for gzip.open function to open the file. I don't know who creates this file, TA or Splunk. I'm going to update the Splunk ticket but do you have any idea what's going on here?

It's just strange that makeresults works but my search doesn't and gives that error.

Thanks!

[guilmxm]

Hi @dpark1113

Hum right, what does the search looks like ? Does if fail with any search besides the makeresults which you confirmed to work ?

None of the errors are related to the addon directly, seems more like a Splunk core failure.

Potentially I would suspect a permissions issue, or context related to your user / roles somehow , test a scheduled search that is owned by admin instead of your owner user or any other user to see if it works.
Those traces would tend to show that the search file artefact called for the alert action couldn't be created or accessed.

Additionally what is the Splunk version, and is it Splunk on premise or Cloud ?

The Splunk addon uses the Python 3 mode, but technically is compatible with both, therefore the issues is happening way before the action is run, so not related to it but to the fact of calling an action apparently

[dpark1113]

Splunk Cloud.

I think I know what's going on and I think this was working before. I'm new to Splunk so it could be from my end.

I'm running a simple search, "sourcetype=ps host=hosta processA". The search doesn't output anything, no event. I want to alert when processA is not running, hence no event.

I think this is causing not creating 'results.srs.zst' file, not sure though. I swear this was working before. After adding "| stats count", all working now.

[guilmxm]

@dpark1113

Generally it's not a good design to create an alert which renders raw events as the final output, rather than something you control in your query.

For example, rather than

index=osnix* user=* action="failure"

You should include your condition as part of the query and end up with a stats format, like in my simplistic example:

index=osnix* user=* action="failure" | stats min(_time) as _time, count by user | where count>10

Basically you want your pipeline to end with something strict and clean, that is reliable.

As far as I am concerned I would suspect that your search was generating too high volume of events, or events were too large, and something was not properly handled.

I understand your issue is fixed 😉

[dpark1113]

Thanks so much @guilmxm!

I'm new to Splunk and I'm learning as issues arise. 🙂

[dpark1113]

I ran the search and I see lot of error messages.

I don't know how this was working yesterday and before. Maybe something got changed in Splunk Cloud.

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - UnboundLocalError: local variable 'rf' referenced before assignment
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - if rf:
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/alert_actions_base.py", line 204, in prepare_meta_for_cam
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - self.prepare_meta_for_cam()
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/ta_jira_service_desk_simple_addon/alert_actions_base.py", line 215, in run
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - exitcode = AlertActionWorkerjira_service_desk("TA-jira-service-desk-simple-addon", "jira_service_desk").run(sys.argv)
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM

05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/etc/apps/TA-jira-service-desk-simple-addon/bin/jira_service_desk.py", line 72, in
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - Traceback (most recent call last):
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - During handling of the above exception, another exception occurred:
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - FileNotFoundError: [Errno 2] No such file or directory: '/opt/splunk/var/run/splunk/dispatch/scheduler_ZWdyYW50QGFzc2V0LWNvbnRyb2wuY29t_VEEtamlyYS1zZXJ2aWNlLWRlc2stc2ltcGxlLWFkZG9u_RMD57a6fb1e9edd3d4b9_at_1590005700_14565/results.srs.zst'
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/lib/python3.7/gzip.py", line 163, in __init_
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - binary_file = GzipFile(filename, gz_mode, compresslevel)
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd
5/20/20
4:16:36.254 PM
05-20-2020 20:16:36.254 +0000 ERROR sendmodalert - action=jira_service_desk STDERR - File "/opt/splunk/lib/python3.7/gzip.py", line 53, in open
host = sh-i-02074304674768d3e.thecompany.splunkcloud.comsource = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd