Why does the splunkd service stop every night on my local Windows PC running Splunk 6.2.2?

[hanshen]

I have Splunk Enterprise installed on my local PC. It's running fine, but splunkd service stops every night. It'sset Automatic with 'Local System' Log ON As.

[hanshen]

Last night splunkd stopped again at 7:50:08 PM 04/14/2015

Faulting application name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Faulting module name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Exception code: 0xc0000005
Fault offset: 0x00e65bb3
Faulting process id: 0x1924
Faulting application start time: 0x01d076e89f074236
Faulting application path: C:\Program Files\Splunk\bin\splunkd.exe
Faulting module path: C:\Program Files\Splunk\bin\splunkd.exe
Report Id: fadb7af9-e300-11e4-a33d-54dbe751310c

[hanshen]

It seems OUT OF MEMORY from splunkd.log last night:

04-14-2015 19:48:58.541 -0400 ERROR STMgr - dir='C:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_505' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
04-14-2015 19:48:58.561 -0400 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
04-14-2015 19:48:58.561 -0400 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
04-14-2015 19:48:58.561 -0400 ERROR STMgr - dir='C:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_505' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
04-14-2015 19:48:58.561 -0400 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
04-14-2015 19:48:58.561 -0400 ERROR STMgr - dir='C:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_505' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
04-14-2015 19:48:58.561 -0400 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index

[dolejh76]

How much memory do you have? Any chance to upgrade memory? To be honest - windows 7 is NOT supported so really I would suggest moving this to a supported OS. I am sure they have there reasons for not supporting it.

[bravon]

I have a simular problem on 2 / 10 index-servers.
Running on Windows 2012 R2
The problem started today.

Faulting application name: splunkd.exe, version: 1538.256.0.48819, time stamp: 0x548a26ed
Faulting module name: splunkd.exe, version: 1538.256.0.48819, time stamp: 0x548a26ed
Exception code: 0xc0000005
Fault offset: 0x0000000000b51bdb
Faulting process id: 0x600
Faulting application start time: 0x01d077566178606b
Faulting application path: D:\Splunk\bin\splunkd.exe
Faulting module path: D:\Splunk\bin\splunkd.exe
Report Id: bed663c0-e34a-11e4-80e1-005056a37962
Faulting package full name: 
Faulting package-relative application ID: 

[bravon]

http://answers.splunk.com/answers/227865/splunkexe-splunkd-crash.html#answer-228904

My problem was that I updated an app (PaloAlto) a couple of days ago and I hadn`t restarted the Windows Servers after the upgrade. After restart the problem started.

[dolejh76]

Roll back and see if it persists? This issue is different and I would open a new question

[dolejh76]

Splunk Enterprise is not supported on Windows 7.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Installation/Systemrequirements

But have you checked the event logs - see if its the same time every night. See what else is happening during that time?

[dolejh76]

Anything in the splunkd log file that points to an issue?

[hanshen]

04-14-2015 11:53:40.846 -0400 ERROR UserManagerPro - Could not get info for non-existent user="nobody"
04-14-2015 11:53:53.328 -0400 ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=C:\Windows\winsxs\x86_microsoft-windows-g..-admfiles.resources_31bf3856ad364e35_6.1.7600.16385_en-us_5eaa9edbbc3c3149\Conf.adml). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
04-13-2015 10:41:18.246 -0400 ERROR FrameworkUtils - Incorrect path to script: .\bin\scripts\usd.py. Script must be located inside $SPLUNK_HOME\bin\scripts.
04-13-2015 10:41:18.246 -0400 ERROR ExecProcessor - Ignoring: ".\bin\scripts\usd.py"
04-13-2015 10:41:18.247 -0400 ERROR FrameworkUtils - Incorrect path to script: .\bin\weather.sh. Script must be located inside $SPLUNK_HOME\bin\scripts.
04-13-2015 10:41:18.247 -0400 ERROR ExecProcessor - Ignoring: ".\bin\weather.sh"
04-13-2015 10:41:36.748 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::subscribeToEvtChannel: Could not subscribe to Windows Event Log channel 'analytic'

[hanshen]

04-13-2015 10:41:36.748 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::subscribeToEvtChannel: Could not subscribe to Windows Event Log channel 'mediafoundationdeviceproxy'
04-13-2015 10:41:36.748 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Init failed, unable to subscribe to Windows Event Log channel 'mediafoundationdeviceproxy':
04-13-2015 10:41:36.748 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::subscribeToEvtChannel: Could not subscribe to Windows Event Log channel 'endpointmapper'
04-13-2015 10:41:36.748 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Init failed, unable to subscribe to Windows Event Log channel 'endpointmapper':
04-13-2015 10:41:36.812 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::subscribeToEvtChannel: Could not subscribe to Windows Event Log channel 'debugchannel'
04-13-2015 10:41:36.812 -0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Init failed, unable to subscribe to Windows Event Log channel 'debugchannel':

[dolejh76]

Assuming you have upgraded to the latest version of Splunk?

[hanshen]

Yes, 6.2.2.

[hanshen]

There are errors in events:
04/13/2015 5:31:03 PM
Faulting application name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Faulting module name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Exception code: 0xc0000005
Fault offset: 0x00e65bb3
Faulting process id: 0xb10
Faulting application start time: 0x01d075f7bd62a575
Faulting application path: C:\Program Files\Splunk\bin\splunkd.exe
Faulting module path: C:\Program Files\Splunk\bin\splunkd.exe
Report Id: 6291005d-e224-11e4-b06b-54dbe751310c

04/10/2015 10:22:29 PM
Faulting application name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb10c6
Exception code: 0xeeab5254
Fault offset: 0x0000812f
Faulting process id: 0xa2c
Faulting application start time: 0x01d073c83a31220d
Faulting application path: C:\Program Files\Splunk\bin\splunkd.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 9995770c-dff1-11e4-b06b-54dbe751310c

[hanshen]

03/27/2015 7:06:37 PM
Faulting application name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Faulting module name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Exception code: 0xc0000005
Fault offset: 0x00e65bb3
Faulting process id: 0xa14
Faulting application start time: 0x01d068ac5d8e8168
Faulting application path: C:\Program Files\Splunk\bin\splunkd.exe
Faulting module path: C:\Program Files\Splunk\bin\splunkd.exe
Report Id: eb3bb943-d4d5-11e4-9c32-54dbe751310c

03/27/2015 1:22:16 AM
Faulting application name: splunkd.exe, version: 1538.512.0.58998, time stamp: 0x54e525a7
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18229, time stamp: 0x51fb10c6
Exception code: 0xeeab5254
Fault offset: 0x0000812f
Faulting process id: 0x2cd4
Faulting application start time: 0x01d06737188e5ac3
Faulting application path: C:\Program Files\Splunk\bin\splunkd.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 3ad6d1bd-d441-11e4-b07b-54dbe751310c

[gyslainlatsa]

hi hanshen,
using what operating system?

[hanshen]

Windows 7 Professional SP1