Monitoring Splunk

Why is splunkd crashing with error "STMgr out of memory failure" at the same time every day since upgrading to Splunk 6.3.1?

philippz
New Member

Since i upgraded our Splunk to 6.3.1 we got the error

"ERROR STMgr - dir='......' out of memory failure". 

The last three days we got this error at the same time and index with the result of crashing of the splunkd.exe.

This happens every day at 05:43 am, and mostly it's the index "defaultdb" that gets an "out of memory failure".

splunkd.log:

11-26-2015 05:43:36.878 +0100 ERROR STMgr - dir='E:\Splunk\defaultdb\db\hot_v1_542' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
11-26-2015 05:43:36.878 +0100 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index

We used Splunk 6.3.1 on Windows Server 2008 RC 2

On the following attachment you can see the occurs of this error over the last 7 days count by the dir.

alt text

0 Karma

lrudolph
Path Finder

Did you get any solution for this from Support? We have the same problem and the whole cluster is crashing almost every day (currently running version 6.3.4.)

0 Karma

lguinn2
Legend

Does your Splunk server meet the specifications for running Splunk?

For Windows, the recommended specifications are
2x six-core, 2+ GHz CPU
12GB RAM
RAID 0 or 1+0
64-bit OS installed

This comes from the Installation manual, under System Requirements

If yes, then the next question is "What searches were running when Splunk crashed?" In particular, I would look at scheduled searches, since you say that this is happening at the same time every day.

FYI, the "defaultdb" is the main index.

Finally, if you have Splunk support, this is the sort of problem that should be reported!

0 Karma

philippz
New Member

Thansk for your anwser.

Yes our Splunk server meet the specifications. This couldn't be the problem.

There is also no scheduled search which could be the reason for the crash.

So i will open a ticket in splunkt support.

thanks

0 Karma

LewisWheeler
Communicator

Did you ever find a solution to this problem?

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...