topic Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"? in All Apps and Add-ons

Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

wangsimingxaxis — Fri, 04 Nov 2016 20:53:56 GMT

Hi I am trying to use Alert Manager in Splunk. Once the alert is triggered and alert manager will do something, it always gets below error. Then there is nothing generated for alert manager data.

Does anyone have any idea ?

"11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -  Traceback (most recent call last):
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 427, in 
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      createIncidentChangeEvent(event, metadata['job_id'], settings.get('index'))
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 157, in createIncidentChangeEvent
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      input.submit(event, hostname = socket.gethostname(), sourcetype = 'incident_change', source = 'alert_handler.py', index=index)
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/lib/python2.7/site-packages/splunk/input.py", line 180, in submit
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      raise splunk.RESTException, (serverResponse.status, msg_text)
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -  splunk.RESTException: [HTTP 400] ["message type=WARN code=None text=supplied index 'alerts' missing;"]
11-04-2016 20:51:01.531 +0000 INFO  sendmodalert - action=alert_manager - Alert action script completed in duration=312 ms with exit code=1
11-04-2016 20:51:01.532 +0000 WARN  sendmodalert - action=alert_manager - Alert action script returned error code=1
11-04-2016 20:51:01.532 +0000 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
11-04-2016 20:51:01.532 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin__launcher__test_at_1478292660_48/results.csv.gz" results_link="http://tbsplunkpeer4.qa1.iad2.xaxis.net:8000/app/launcher/@go?sid=scheduler__admin__launcher__test_at_1478292660_48"'
"

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Simon — Fri, 04 Nov 2016 22:13:57 GMT

Did you configure an index? By default, the Alert Manager uses "alerts". If you wish to use another one, please configure it in the App General settings.
Note: The index definition also has to exist on the searchead as the Splunk REST API isn't aware of indexes only existing on indexers

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

wangsimingxaxis — Sat, 05 Nov 2016 13:07:21 GMT

thanks for the reply, I think I use the default setting , index="alerts" , which I confirmed in the App General Settings.

And I cannot find any useful documents on website to guide me how to setup Alert Manager ~~~

PS: I run this alert manager on my testing standalone splunk. thus no need to worry about indexer, search head, master communication and sync issue.

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Simon — Mon, 07 Nov 2016 21:18:14 GMT

Did you configure the index in Splunk as well? Go to "Settings" -> "Indexes".

By the way, we have a new docs page here: http://docs.alertmanager.info

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

wangsimingxaxis — Mon, 07 Nov 2016 21:29:03 GMT

Thats helped much better ~~~ thx the issue is resolved

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

wangsimingxaxis — Tue, 29 Sep 2020 11:43:46 GMT

Hi Alert Manager Support,
Sorry to bothering you again.

I tested it successfully in our Dev env, but when I move on to QA env. I got below errors which I never see before .

11-09-2016 20:18:25.385 +0000 INFO  sendmodalert - Invoking modular alert action=alert_manager for search="testtesttesttest" sid="scheduler__admin__xaxis__testtesttesttest_at_1478722680_16" in app="xaxis" owner="admin" type="saved"
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -  Traceback (most recent call last):
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 363, in 
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -      savedSearch = getSavedSearch(payload.get('app'), search_name, sessionKey)
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 288, in getSavedSearch
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -      savedSearch = getRestData(uri, sessionKey)
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 263, in getRestData
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -      serverResponse, serverContent = rest.simpleRequest(uri, sessionKey=sessionKey, getargs={'output_mode': 'json'})
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 534, in simpleRequest
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -      raise splunk.ResourceNotFound, uri
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -  splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/xaxis/admin/savedsearch/testtesttesttest?output_mode=json
11-09-2016 20:18:25.548 +0000 INFO  sendmodalert - action=alert_manager - Alert action script completed in duration=162 ms with exit code=1
11-09-2016 20:18:25.548 +0000 WARN  sendmodalert - action=alert_manager - Alert action script returned error code=1
11-09-2016 20:18:25.548 +0000 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
11-09-2016 20:18:25.549 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin__xaxis__testtesttesttest_at_1478722680_16/results.csv.gz" results_link="http://tbsplunksearch1.qa1.iad2.xaxis.net:8000/app/xaxis/@go?sid=scheduler__admin__xaxis__testtesttesttest_at_1478722680_16"

Any suggestions? appreciate it

PS: my working steps:
on Search head :
1, install alert-manager_214.tgz
2, install TA-alert_manager.tar.gz
3, create index=alerts

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Simon — Wed, 09 Nov 2016 20:42:47 GMT

Check the permissions of your saved search. They must be shared on app level at least. It looks like your search is private.