Why is Pagerduty integration with Splunk not working and returning the following error code?: "Error in 'sendalert' command: Alert script returned error code 1"

[bishtk]

Dear All,

We have Prod and DR environment. And the Splunk search head of production setup is working well with Pagerduty. But, the DR setup is not. I even tried to replace Pagerduty integration URL of DR setup with Prod setup but still got same errors. Please suggest.
Pagerduty_incidents version is 1.1
splunk enterprise v 6.5.1
Below are the errors:

Time    Event 
9/19/18 
8:05:04.415 AM  
09-19-2018 08:05:04.415 -0400 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert pagerduty results_file="$SPLUNK_HOME/var/run/splunk/dispatch/scheduler__bishtk_d2Vic2l0ZV9tb25pdG9yaW5n__RMD5d360115fdafa9e4a_at_1537358700_29811/results.csv.gz" results_link="http://:8000/app/website_monitoring/@go?sid=scheduler__bishtk_d2Vic2l0ZV9tb25pdG9yaW5n__RMD5d360115fdafa9e4a_at_1537358700_29811"' 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.415 AM  
09-19-2018 08:05:04.415 -0400 WARN sendmodalert - action=pagerduty - Alert action script returned error code=1 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.415 AM  
09-19-2018 08:05:04.415 -0400 INFO sendmodalert - action=pagerduty - Alert action script completed in duration=72 ms with exit code=1 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.409 AM  
09-19-2018 08:05:04.409 -0400 ERROR sendmodalert - action=pagerduty STDERR - TypeError: object of type 'NoneType' has no len() 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.409 AM  
09-19-2018 08:05:04.409 -0400 ERROR sendmodalert - action=pagerduty STDERR - if len(url) == 32: 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.409 AM  
09-19-2018 08:05:04.409 -0400 ERROR sendmodalert - action=pagerduty STDERR - File "$SPLUNK_HOME/etc/apps/pagerduty_incidents/bin/pagerduty.py", line 18, in send_notification 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.409 AM  
09-19-2018 08:05:04.409 -0400 ERROR sendmodalert - action=pagerduty STDERR - success = send_notification(payload) 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.409 AM  
09-19-2018 08:05:04.409 -0400 ERROR sendmodalert - action=pagerduty STDERR - File "$SPLUNK_HOME/etc/apps/pagerduty_incidents/bin/pagerduty.py", line 43, in  
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.409 AM  
09-19-2018 08:05:04.409 -0400 ERROR sendmodalert - action=pagerduty STDERR - Traceback (most recent call last): 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
8:05:04.340 AM  
09-19-2018 08:05:04.340 -0400 INFO sendmodalert - Invoking modular alert action=pagerduty for search="SEV3::PROD Website Monitoring Alert for DNS" sid="scheduler__bishtk_d2Vic2l0ZV9tb25pdG9yaW5n__RMD5d360115fdafa9e4a_at_1537358700_29811" in app="website_monitoring" owner="bishtk" type="saved" 
host =  <Splunksearchheadhostname> source = $SPLUNK_HOME/var/log/splunk/splunkd.log sourcetype = splunkd 
9/19/18 
7:00:08.168 AM  
09-19-2018 07:00:08.168 -0400 ERROR SearchScheduler - Error in 'sendalert' command: Al

Thanks in advance,

[pdehlke]

Coming in here, years later, to document this for anyone who comes across it.

The pagerduty app requires that the user who owns the alert must have, at a minimum, the list_storage_passwords capability. Almost certainly this was the issue here.

[bishtk]

Any suggestion guys on this issue?

Thanks,
Kundan B.Bisht