hello
recently my Splunk not start, it happens suddenly,after i notice splunk web not work,login to windows server and see it crash and have auto restart,after that i start splunk but get this :
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Failed to determine if running as service user: LookupAccountName: No mapping between account names and security IDs was done.
(skipping validation of index paths because not running as ASADC\Mediterranean)
Validated: _audit _internal _introspection _telemetry _thefishbucket history main msad mssql perfmon summary vmware-esxilog vmware-inv vmware-perf vmware-taskevent vmware-vclog windows wineventlog winevents
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from 'C:\Program Files\Splunk\splunk-7.1.2-a0c72a66db66-windows-64-manifest'
File 'C:\Program Files\Splunk\etc/system/default/indexes.conf' changed.
File 'C:\Program Files\Splunk\etc/system/default/inputs.conf' changed.
File 'C:\Program Files\Splunk\etc/system/default/limits.conf' changed.
Problems were found, please review your files and move customizations to local
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Splunkd: Stopped
what can i do?i chek log file of splunk and fined this :
10-26-2019 08:02:54.889 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.904 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.904 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetFullServerPath: Failed to bind to root 'LDAP://pri02.eng.ad.splunk.com/rootDSE': err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetFullServerPath: Failed to bind to root 'LDAP://pri01.eng.ad.splunk.com/rootDSE': err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetServerPath: Failed to bind to root: err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::GetDCAttributes: Failed to get AD server path.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed: (0x80004005)Unspecified error -- attempting to reload server path
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetServerPath: Failed to bind to root: err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::GetDCAttributes: Failed to get AD server path.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed: (0x80004005)Unspecified error -- attempting to reload server path
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdQuery::OutputStartEvent: Failed to search attributes of root object: err='0xa'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::OutputStartEvent: Failed in OutputStartEvent,
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed again with DCName='Asa-Dc.AsaDc.local': (0x80004005)Unspecified error -- no more retries
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitor::init: Failed to initialize Active Directory usn context.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitorThread::launchADMonitor: Failed to initialize ADMonitor='admon://SecondTargetDC', targedDC='pri02.eng.ad.splunk.com'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdQuery::OutputStartEvent: Failed to search attributes of root object: err='0xa'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::OutputStartEvent: Failed in OutputStartEvent,
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed again with DCName='Asa-Dc.AsaDc.local': (0x80004005)Unspecified error -- no more retries
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitor::init: Failed to initialize Active Directory usn context.
... View more