All Apps and Add-ons
Highlighted

Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Hi I am trying to use Alert Manager in Splunk. Once the alert is triggered and alert manager will do something, it always gets below error. Then there is nothing generated for alert manager data.

Does anyone have any idea ?

"11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -  Traceback (most recent call last):
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 427, in 
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      createIncidentChangeEvent(event, metadata['job_id'], settings.get('index'))
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 157, in createIncidentChangeEvent
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      input.submit(event, hostname = socket.gethostname(), sourcetype = 'incident_change', source = 'alert_handler.py', index=index)
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/lib/python2.7/site-packages/splunk/input.py", line 180, in submit
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      raise splunk.RESTException, (serverResponse.status, msg_text)
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -  splunk.RESTException: [HTTP 400] ["message type=WARN code=None text=supplied index 'alerts' missing;"]
11-04-2016 20:51:01.531 +0000 INFO  sendmodalert - action=alert_manager - Alert action script completed in duration=312 ms with exit code=1
11-04-2016 20:51:01.532 +0000 WARN  sendmodalert - action=alert_manager - Alert action script returned error code=1
11-04-2016 20:51:01.532 +0000 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
11-04-2016 20:51:01.532 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin__launcher__test_at_1478292660_48/results.csv.gz" results_link="http://tbsplunkpeer4.qa1.iad2.xaxis.net:8000/app/launcher/@go?sid=scheduler__admin__launcher__test_at_1478292660_48"'
"
0 Karma
Highlighted

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Contributor

Did you configure an index? By default, the Alert Manager uses "alerts". If you wish to use another one, please configure it in the App General settings.
Note: The index definition also has to exist on the searchead as the Splunk REST API isn't aware of indexes only existing on indexers

View solution in original post

Highlighted

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

thanks for the reply, I think I use the default setting , index="alerts" , which I confirmed in the App General Settings.

And I cannot find any useful documents on website to guide me how to setup Alert Manager ~~~

PS: I run this alert manager on my testing standalone splunk. thus no need to worry about indexer, search head, master communication and sync issue.

0 Karma
Highlighted

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Contributor

Did you configure the index in Splunk as well? Go to "Settings" -> "Indexes".

By the way, we have a new docs page here: http://docs.alertmanager.info

Highlighted

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Thats helped much better ~~~ thx the issue is resolved

0 Karma
Highlighted

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Hi Alert Manager Support,
Sorry to bothering you again.

I tested it successfully in our Dev env, but when I move on to QA env. I got below errors which I never see before .

11-09-2016 20:18:25.385 +0000 INFO  sendmodalert - Invoking modular alert action=alert_manager for search="testtesttesttest" sid="scheduler__admin__xaxis__testtesttesttest_at_1478722680_16" in app="xaxis" owner="admin" type="saved"
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -  Traceback (most recent call last):
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 363, in 
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -      savedSearch = getSavedSearch(payload.get('app'), search_name, sessionKey)
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 288, in getSavedSearch
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -      savedSearch = getRestData(uri, sessionKey)
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 263, in getRestData
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -      serverResponse, serverContent = rest.simpleRequest(uri, sessionKey=sessionKey, getargs={'output_mode': 'json'})
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 534, in simpleRequest
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -      raise splunk.ResourceNotFound, uri
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -  splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/xaxis/admin/savedsearch/testtesttesttest?output_mode=json
11-09-2016 20:18:25.548 +0000 INFO  sendmodalert - action=alert_manager - Alert action script completed in duration=162 ms with exit code=1
11-09-2016 20:18:25.548 +0000 WARN  sendmodalert - action=alert_manager - Alert action script returned error code=1
11-09-2016 20:18:25.548 +0000 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
11-09-2016 20:18:25.549 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin__xaxis__testtesttesttest_at_1478722680_16/results.csv.gz" results_link="http://tbsplunksearch1.qa1.iad2.xaxis.net:8000/app/xaxis/@go?sid=scheduler__admin__xaxis__testtesttesttest_at_1478722680_16"

Any suggestions? appreciate it

PS: my working steps:
on Search head :
1, install alert-manager214.tgz
2, install TA-alert
manager.tar.gz
3, create index=alerts

0 Karma
Highlighted

Re: Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

Contributor

Check the permissions of your saved search. They must be shared on app level at least. It looks like your search is private.

0 Karma