All Apps and Add-ons

Alert Manager: How to resolve multiple errors received such as "Error in 'sendalert' command: Alert script returned error code 1"?

wangsimingxaxis
Explorer

Hi I am trying to use Alert Manager in Splunk. Once the alert is triggered and alert manager will do something, it always gets below error. Then there is nothing generated for alert manager data.

Does anyone have any idea ?

"11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -  Traceback (most recent call last):
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 427, in 
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      createIncidentChangeEvent(event, metadata['job_id'], settings.get('index'))
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 157, in createIncidentChangeEvent
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      input.submit(event, hostname = socket.gethostname(), sourcetype = 'incident_change', source = 'alert_handler.py', index=index)
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/lib/python2.7/site-packages/splunk/input.py", line 180, in submit
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -      raise splunk.RESTException, (serverResponse.status, msg_text)
11-04-2016 20:51:01.519 +0000 ERROR sendmodalert - action=alert_manager STDERR -  splunk.RESTException: [HTTP 400] ["message type=WARN code=None text=supplied index 'alerts' missing;"]
11-04-2016 20:51:01.531 +0000 INFO  sendmodalert - action=alert_manager - Alert action script completed in duration=312 ms with exit code=1
11-04-2016 20:51:01.532 +0000 WARN  sendmodalert - action=alert_manager - Alert action script returned error code=1
11-04-2016 20:51:01.532 +0000 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
11-04-2016 20:51:01.532 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin__launcher__test_at_1478292660_48/results.csv.gz" results_link="http://tbsplunkpeer4.qa1.iad2.xaxis.net:8000/app/launcher/@go?sid=scheduler__admin__launcher__test_at_1478292660_48"'
"
0 Karma
1 Solution

Simon
Contributor

Did you configure an index? By default, the Alert Manager uses "alerts". If you wish to use another one, please configure it in the App General settings.
Note: The index definition also has to exist on the searchead as the Splunk REST API isn't aware of indexes only existing on indexers

View solution in original post

Simon
Contributor

Did you configure an index? By default, the Alert Manager uses "alerts". If you wish to use another one, please configure it in the App General settings.
Note: The index definition also has to exist on the searchead as the Splunk REST API isn't aware of indexes only existing on indexers

Simon
Contributor

Check the permissions of your saved search. They must be shared on app level at least. It looks like your search is private.

0 Karma

wangsimingxaxis
Explorer

Hi Alert Manager Support,
Sorry to bothering you again.

I tested it successfully in our Dev env, but when I move on to QA env. I got below errors which I never see before .

11-09-2016 20:18:25.385 +0000 INFO  sendmodalert - Invoking modular alert action=alert_manager for search="testtesttesttest" sid="scheduler__admin__xaxis__testtesttesttest_at_1478722680_16" in app="xaxis" owner="admin" type="saved"
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -  Traceback (most recent call last):
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 363, in 
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -      savedSearch = getSavedSearch(payload.get('app'), search_name, sessionKey)
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 288, in getSavedSearch
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -      savedSearch = getRestData(uri, sessionKey)
11-09-2016 20:18:25.537 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 263, in getRestData
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -      serverResponse, serverContent = rest.simpleRequest(uri, sessionKey=sessionKey, getargs={'output_mode': 'json'})
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -    File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 534, in simpleRequest
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -      raise splunk.ResourceNotFound, uri
11-09-2016 20:18:25.538 +0000 ERROR sendmodalert - action=alert_manager STDERR -  splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/xaxis/admin/savedsearch/testtesttesttest?output_mode=json
11-09-2016 20:18:25.548 +0000 INFO  sendmodalert - action=alert_manager - Alert action script completed in duration=162 ms with exit code=1
11-09-2016 20:18:25.548 +0000 WARN  sendmodalert - action=alert_manager - Alert action script returned error code=1
11-09-2016 20:18:25.548 +0000 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
11-09-2016 20:18:25.549 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__admin__xaxis__testtesttesttest_at_1478722680_16/results.csv.gz" results_link="http://tbsplunksearch1.qa1.iad2.xaxis.net:8000/app/xaxis/@go?sid=scheduler__admin__xaxis__testtesttesttest_at_1478722680_16"

Any suggestions? appreciate it

PS: my working steps:
on Search head :
1, install alert-manager_214.tgz
2, install TA-alert_manager.tar.gz
3, create index=alerts

0 Karma

wangsimingxaxis
Explorer

thanks for the reply, I think I use the default setting , index="alerts" , which I confirmed in the App General Settings.

And I cannot find any useful documents on website to guide me how to setup Alert Manager ~~~

PS: I run this alert manager on my testing standalone splunk. thus no need to worry about indexer, search head, master communication and sync issue.

0 Karma

Simon
Contributor

Did you configure the index in Splunk as well? Go to "Settings" -> "Indexes".

By the way, we have a new docs page here: http://docs.alertmanager.info

wangsimingxaxis
Explorer

Thats helped much better ~~~ thx the issue is resolved

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...