Since i upgraded our Splunk to 6.3.1 we got the error
"ERROR STMgr - dir='......' out of memory failure".
The last three days we got this error at the same time and index with the result of crashing of the splunkd.exe.
This happens every day at 05:43 am, and mostly it's the index "defaultdb" that gets an "out of memory failure".
11-26-2015 05:43:36.878 +0100 ERROR STMgr - dir='E:\Splunk\defaultdb\db\hot_v1_542' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start 11-26-2015 05:43:36.878 +0100 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
We used Splunk 6.3.1 on Windows Server 2008 RC 2
On the following attachment you can see the occurs of this error over the last 7 days count by the dir.
Does your Splunk server meet the specifications for running Splunk?
For Windows, the recommended specifications are
2x six-core, 2+ GHz CPU
RAID 0 or 1+0
64-bit OS installed
This comes from the Installation manual, under System Requirements
If yes, then the next question is "What searches were running when Splunk crashed?" In particular, I would look at scheduled searches, since you say that this is happening at the same time every day.
FYI, the "defaultdb" is the main index.
Finally, if you have Splunk support, this is the sort of problem that should be reported!
Thansk for your anwser.
Yes our Splunk server meet the specifications. This couldn't be the problem.
There is also no scheduled search which could be the reason for the crash.
So i will open a ticket in splunkt support.