splunk not start

[givehchin]

hello
recently my Splunk not start, it happens suddenly,after i notice splunk web not work,login to windows server and see it crash and have auto restart,after that i start splunk but get this :
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Failed to determine if running as service user: LookupAccountName: No mapping between account names and security IDs was done.
(skipping validation of index paths because not running as ASADC\Mediterranean)
Validated: _audit _internal _introspection _telemetry _thefishbucket history main msad mssql perfmon summary vmware-esxilog vmware-inv vmware-perf vmware-taskevent vmware-vclog windows wineventlog winevents
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from 'C:\Program Files\Splunk\splunk-7.1.2-a0c72a66db66-windows-64-manifest'
File 'C:\Program Files\Splunk\etc/system/default/indexes.conf' changed.
File 'C:\Program Files\Splunk\etc/system/default/inputs.conf' changed.
File 'C:\Program Files\Splunk\etc/system/default/limits.conf' changed.
Problems were found, please review your files and move customizations to local
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Splunkd: Stopped

what can i do?i chek log file of splunk and fined this :
10-26-2019 08:02:54.889 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.904 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.904 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetFullServerPath: Failed to bind to root 'LDAP://pri02.eng.ad.splunk.com/rootDSE': err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetFullServerPath: Failed to bind to root 'LDAP://pri01.eng.ad.splunk.com/rootDSE': err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetServerPath: Failed to bind to root: err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::GetDCAttributes: Failed to get AD server path.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed: (0x80004005)Unspecified error -- attempting to reload server path
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetServerPath: Failed to bind to root: err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::GetDCAttributes: Failed to get AD server path.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed: (0x80004005)Unspecified error -- attempting to reload server path
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdQuery::OutputStartEvent: Failed to search attributes of root object: err='0xa'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::OutputStartEvent: Failed in OutputStartEvent,
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed again with DCName='Asa-Dc.AsaDc.local': (0x80004005)Unspecified error -- no more retries
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitor::init: Failed to initialize Active Directory usn context.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitorThread::launchADMonitor: Failed to initialize ADMonitor='admon://SecondTargetDC', targedDC='pri02.eng.ad.splunk.com'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdQuery::OutputStartEvent: Failed to search attributes of root object: err='0xa'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::OutputStartEvent: Failed in OutputStartEvent,
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed again with DCName='Asa-Dc.AsaDc.local': (0x80004005)Unspecified error -- no more retries
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitor::init: Failed to initialize Active Directory usn context.

[givehchin · Accepted solution]

We solve it, just restart Splunk, Splunk web services with local administrator privileges

[woodcock]

Great! Now come back and click Accept on your answer to close the question and UpVote any other answers or comments that helped you.

[woodcock]

It looks like Splunk is complaining about the Index values defined on your D drive. Is your D drive mounted? Is your D drive full? If so, fix that and it shoud be OK. The long term solution is to stop running your Splunk infrastructure (Indexers) on Windows OS:
https://answers.splunk.com/answers/516059/what-are-the-pain-points-with-deploying-your-splun.html

[givehchin]

I checked Splunk version from Splunk.version and splunk.exe,they show my current version 7.1.2 wich it mean not updated at all

[clintonburnett]

Several issues that could be related.
Seems like could be a permissions issue of the user permissions Splunk as a service is running as
I would check the password for the splunkd service if not using managed service accounts. The two errors listed below points to this.

Skipping validation of index paths error due to not running as ASADC\Mediterranean
Failed to determine if running as service user: LookupAccountName: No mapping between account names and security IDs was done.

Another possibility looks like could be a GPO error and example of a fix for this is below. Not sure what version of windows server that is running.
http://www.rebeladmin.com/2016/01/how-to-fix-error-no-mapping-between-account-names-and-security-ids...

Another issue that could cause problems is placing changes in default not local directory - will cause issues anytime you update Splunk because changes will be overwritten.

[givehchin]

that error(mediteranian user) probably be there in past couple months

I using Windows Server 2016

does Splunk can automatically update?

[woodcock]

Did you upgrade to v8.0?

[givehchin]

no, I do not upgrade it, can Splunk automatically update???

[woodcock]

No, it cannot.

[vsai0718]

What kind of instance is it. If it is an EC2, try stopping and starting the instance. Then go to the terminal and start the splunk.

[givehchin]

I cant understand, what is EC2??? I stop all Splunk in task manager and start it again, nothing changed