07-22-2011 09:28:08.124 INFO databasePartitionPolicy - Found timestamp file ! at /opt/splunk/var/lib/splunk/os/db/CreationTime 07-22-2011 09:28:08.127 INFO databasePartitionPolicy - CREATION TIME for /opt/splunk/var/lib/splunk/os/db : 1298053588 07-22-2011 09:28:08.127 INFO databasePartitionPolicy - opening database /opt/splunk/var/lib/splunk/os/db/hot_v1_0 07-22-2011 09:28:08.127 INFO timeinvertedIndex - Opening /opt/splunk/var/lib/splunk/os/db/hot_v1_0 07-22-2011 09:28:08.128 INFO timeinvertedIndex - No files to decompress on create 07-22-2011 09:28:08.128 INFO timeinvertedIndex - create by dirname /opt/splunk/var/lib/splunk/os/db/hot_v1_0 07-22-2011 09:28:08.128 ERROR WordPositionData - couldn't parse hash code: 07-22-2011 09:28:08.133 WARN MetaData - found error in metadata file, attempting recovery: /opt/splunk/var/lib/splunk/os/db/ hot_v1_0/Hosts.data 07-22-2011 09:28:08.134 WARN MetaData - /opt/splunk/var/lib/splunk/os/db/hot_v1_0/Hosts.data: initiating recover attempt
You should probably take a look at this:
'splunk clean all' may have resolved the problem, but its because it deleted all of your indexed data. For most people, this is not an acceptable solution. The metadata in one of the indexes has probably become corrupt, and that is the reason clean all worked. It seems a bit of an extreme response to the situation, but I am glad to hear things are working again.