I'm thinking index corruption, right?

[grijhwani]

Search =

index=index_root*| stats first(_time) as latest last(_time) as earliest count(index) by index | convert timeformat="%Y-%m-%d %H:%M:%S" ctime(earliest) ctime(latest)

Eventually results in

"[{{hostname}}] Error in 'IndexScopedSearch': The search failed. More than 3125000 events found at time 1338306857."

The epoc time of the error (2012-05-29 15:54:17 BST) is 4 seconds earlier than the earliest reported time before it bombs (although it reports it erroneously as 16:54:21). If I tie the search down to the last few seconds only 11 events are reported for that particular second (:21).

[Kate_Lawrence-G]

kind of...you basically have millions of events with the same date/time stamp which is causing an logical error in Splunk as to how many events it can process per second. (see http://splunk-base.splunk.com/answers/2958/error-in-indexscopedsearch-the-search-failed for very good explanation)

This is usually related to a bad time date stamp logic in a props.conf somewhere that isn't breaking the events down correctly.

[grijhwani]

The thing is I can be pretty certain that in the original data this was not the case, and that something has intervened in the index. I'm going back auditing historic data that were in use as live data at the time, and no-one flagged up a fault back then.