Installation

splunk not start

givehchin
Path Finder

hello
recently my Splunk not start, it happens suddenly,after i notice splunk web not work,login to windows server and see it crash and have auto restart,after that i start splunk but get this :
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Failed to determine if running as service user: LookupAccountName: No mapping between account names and security IDs was done.
(skipping validation of index paths because not running as ASADC\Mediterranean)
Validated: _audit _internal _introspection _telemetry _thefishbucket history main msad mssql perfmon summary vmware-esxilog vmware-inv vmware-perf vmware-taskevent vmware-vclog windows wineventlog winevents
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from 'C:\Program Files\Splunk\splunk-7.1.2-a0c72a66db66-windows-64-manifest'
File 'C:\Program Files\Splunk\etc/system/default/indexes.conf' changed.
File 'C:\Program Files\Splunk\etc/system/default/inputs.conf' changed.
File 'C:\Program Files\Splunk\etc/system/default/limits.conf' changed.
Problems were found, please review your files and move customizations to local
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Splunkd: Stopped

what can i do?i chek log file of splunk and fined this :
10-26-2019 08:02:54.889 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.904 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.904 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.904 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.920 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.920 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.935 +0330 ERROR STMgr - dir='D:\Warm\defaultdb\db\hot_v1_10953' out of memory failure rc=1 warm_rc[-2,8] from st_txn_start
10-26-2019 08:02:54.935 +0330 ERROR StreamGroup - unexpected rc=1 from IndexableValue->index
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetFullServerPath: Failed to bind to root 'LDAP://pri02.eng.ad.splunk.com/rootDSE': err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetFullServerPath: Failed to bind to root 'LDAP://pri01.eng.ad.splunk.com/rootDSE': err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetServerPath: Failed to bind to root: err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::GetDCAttributes: Failed to get AD server path.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed: (0x80004005)Unspecified error -- attempting to reload server path
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADGetServerPath: Failed to bind to root: err='0x8007203a' - 'The server is not operational.'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::GetDCAttributes: Failed to get AD server path.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed: (0x80004005)Unspecified error -- attempting to reload server path
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdQuery::OutputStartEvent: Failed to search attributes of root object: err='0xa'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::OutputStartEvent: Failed in OutputStartEvent,
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed again with DCName='Asa-Dc.AsaDc.local': (0x80004005)Unspecified error -- no more retries
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitor::init: Failed to initialize Active Directory usn context.
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitorThread::launchADMonitor: Failed to initialize ADMonitor='admon://SecondTargetDC', targedDC='pri02.eng.ad.splunk.com'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdQuery::OutputStartEvent: Failed to search attributes of root object: err='0xa'
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::OutputStartEvent: Failed in OutputStartEvent,
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - AdEventCollector::InitCollector: LoadContextState failed again with DCName='Asa-Dc.AsaDc.local': (0x80004005)Unspecified error -- no more retries
10-26-2019 08:02:54.967 +0330 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-admon.exe"" splunk-admon - ADMonitor::init: Failed to initialize Active Directory usn context.

0 Karma
1 Solution

givehchin
Path Finder

We solve it, just restart Splunk, Splunk web services with local administrator privileges

View solution in original post

0 Karma

givehchin
Path Finder

We solve it, just restart Splunk, Splunk web services with local administrator privileges

0 Karma

woodcock
Esteemed Legend

Great! Now come back and click Accept on your answer to close the question and UpVote any other answers or comments that helped you.

0 Karma

woodcock
Esteemed Legend

It looks like Splunk is complaining about the Index values defined on your D drive. Is your D drive mounted? Is your D drive full? If so, fix that and it shoud be OK. The long term solution is to stop running your Splunk infrastructure (Indexers) on Windows OS:
https://answers.splunk.com/answers/516059/what-are-the-pain-points-with-deploying-your-splun.html

0 Karma

givehchin
Path Finder

I checked Splunk version from Splunk.version and splunk.exe,they show my current version 7.1.2 wich it mean not updated at all

0 Karma

clintonburnett
Explorer

Several issues that could be related.
Seems like could be a permissions issue of the user permissions Splunk as a service is running as
I would check the password for the splunkd service if not using managed service accounts. The two errors listed below points to this.

Skipping validation of index paths error due to not running as ASADC\Mediterranean
Failed to determine if running as service user: LookupAccountName: No mapping between account names and security IDs was done.

Another possibility looks like could be a GPO error and example of a fix for this is below. Not sure what version of windows server that is running.
http://www.rebeladmin.com/2016/01/how-to-fix-error-no-mapping-between-account-names-and-security-ids...

Another issue that could cause problems is placing changes in default not local directory - will cause issues anytime you update Splunk because changes will be overwritten.

0 Karma

givehchin
Path Finder

that error(mediteranian user) probably be there in past couple months

I using Windows Server 2016

does Splunk can automatically update?

0 Karma

woodcock
Esteemed Legend

Did you upgrade to v8.0?

0 Karma

givehchin
Path Finder

no, I do not upgrade it, can Splunk automatically update???

0 Karma

woodcock
Esteemed Legend

No, it cannot.

0 Karma

vsai0718
Path Finder

What kind of instance is it. If it is an EC2, try stopping and starting the instance. Then go to the terminal and start the splunk.

0 Karma

givehchin
Path Finder

I cant understand, what is EC2??? I stop all Splunk in task manager and start it again, nothing changed

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...