Archive
Highlighted

Indexer not responding.. Replication status is 'Failed'

New Member

A reboot cured the above issue( In title), which is far from ideal.

See the below lines logged in 'Splunkd.log' on the indexer before reboot of the server.

03-11-2013 17:31:04.383 +0000 WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded
03-11-2013 17:31:12.973 +0000 ERROR ExecProcessor - message from "python /opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py" Traceback (most recent call last):
03-11-2013 17:31:12.973 +0000 ERROR ExecProcessor - message from "python /opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py" File "/opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py", line 161, in
03-11-2013 17:31:12.973 +0000 ERROR ExecProcessor - message from "python /opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py" next(token)
03-11-2013 17:31:12.973 +0000 ERROR ExecProcessor - message from "python /opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py" File "/opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py", line 67, in next
03-11-2013 17:31:12.973 +0000 ERROR ExecProcessor - message from "python /opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py" en = search(token, srch='status=1')

And a snippet after reboot

03-12-2013 13:47:22.800 +0000 ERROR ExecProcessor - message from "python /opt/splunkuat/etc/apps/SplunkDeploymentMonitor-old1/bin/scriptedinputs/dmbackfillfactory.py" splunk.ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/SplunkDeploymentMonitor/backfill/dm_backfill?sort_key=seed&...; [{'text': 'Application is disabled: SplunkDeploymentMonitor', 'code': None, 'type': 'ERROR'}]

Does anyone know whether the above errors can cause Indexer to slow down and replication status move to 'failed' state?
Thanks

Tags (1)
0 Karma
Highlighted

Re: Indexer not responding.. Replication status is 'Failed'

Explorer

I had the same issue but instead of rebooting I just disabled the search peer and reenabled it, after that I got a successful replciation. Other than that I don't know what cause the issue.

0 Karma
Highlighted

Re: Indexer not responding.. Replication status is 'Failed'

Engager

Updating this thread with as I had the same issue. There was a large csv file located in the lookup folder in one of my apps, which was causing the replication bundle to balloon. Removing this file resolved the issue.

0 Karma
Highlighted

Re: Indexer not responding.. Replication status is 'Failed'

Path Finder

If you have huge csv's in the apps , you will see these kind of issues .
you need to disable search peers and enable back , If you still see the issue need to disable few apps on the SH .

0 Karma