During startup recovery, error reading from process runner child: Bad file number. Splunk fsck failed with error code '8'.

[zliu]

When startup and recovering from a unclean shutdown.
Perform recovery now? [y/n] y
Recovering (across all data)...
Error reading from process runner child: Bad file number
Splunk fsck failed with error code '8'. Please file a case online at http://www.splunk.com/page/submit_issue

running splunk as a heavy forwarder without indexing on a some AIX systems with a non root user called splkadm.

Splunk 4.2.1

[Rob · Accepted solution]

This is an issue specific to AIX causing the fsck utility to fail.

If you do not wish to see this error you may wish to delete the meta.dirty file that is located in the $SPLUNK_HOME/var/lib/splunk/defaultdb/db/ directory.

Keep in mind that this will only delete the file that was generated on an unclean shutdown and does not actually repair the buckets. To do so you may wish to run the fsck utility manually with the following command:

$SPLUNK_HOME/bin/splunk cmd splunkd fsck --all --mode metadata --repair

Otherwise, have a look at the following answer which describes how to validate the metadata files as well as the bucket tsidx files.

http://splunk-base.splunk.com/answers/5374/how-to-quickly-validate-the-metadata-files-of-a-given-ind...

[zliu]

It could be caused by Splunk on AIX 6.x.
Splunk is not certified to work on AIX 6.x.