Failing manual Splunk-optimize when 'The index processor has paused data flow. Too many tsidx files' with erno 12

[chrzz]

Hello

I've started to get this error message:

The index processor has paused data flow. Too many tsidx files in idx=_audit bucket="/opt/splunk/var/lib/splunk/audit/db/hot_v1_13" , waiting for the splunk-optimize indexing helper to catch up merging them. Ensure reasonable disk space is available, and that I/O write throughput is not compromised.

I then tried the manual splunk-optimize, but that returned this error message:

tm= 1568090447 ERROR merge failed for path=/opt/splunk/var/lib/splunk/audit/db/hot_v1_13 rc=-2 wrc=-2 errno=12 file=/opt/splunk/var/lib/splunk/audit/db/hot_v1_13/1567134306-1567134305-16403447236531428423.tsidx hint=_init_reader_helper in _merge_all_postings_n]
tm= 1568090447 ERROR optimize finished: failed, see rc for more details, dir=/opt/splunk/var/lib/splunk/audit/db/hot_v1_13, rc=-2 (unsigned 254), errno=12
tm= 1568090447 INFO  exiting splunk-optimize process with rc=-2 (unsigned 254)

I've tried to search for "errno=12", but I can't find any info regarding it (just other error numbers).

All my indexes have default settings.

Any suggestions?

[GalaxySplunker]

Hello,

I'm encountering the exact same error, only for another index. And it happens a lot, nearly on a daily basis (work days)

Is there any new update on this topic please ? 

Thank you for your help

[lapplander · Accepted solution]

If you run into a situation of too many tsidx files and splunk can't resolve it by it self. Try restarting splunk with splunk restart Upon starting splunk again, all hot buckets will be rolled to warm and the tsidx files gets merged into fewer files.

Watch the stdout from the restart comand to see if it throws any errors and review $SPLUNK_HOME/var/log/splunk/splunkd.log and lookout for any WARN or ERROR messages following the last shutdown.