topic Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3" in Monitoring Splunk

Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3"

willsy — Mon, 10 Dec 2018 18:43:02 GMT

I restarted my server, and the Splunk web GUI didn't load up. My other servers and search heads load up, just not this particular search head. I know the issue is meant to be multiple indexes of the same thing, but I can't seem to see which one would be the problem child. This is the error i continually get every time i try, and either manually restart splunk services or restart the machine again.

12-10-2018 15:35:27.962 +0000 INFO  loader - win-service: Starting as a Windows service: will run various system checks first...
12-10-2018 15:35:27.962 +0000 INFO  loader - win-service: Splunk starting as a local administrator
12-10-2018 15:35:27.962 +0000 INFO  loader - Automatic migration of modular inputs
12-10-2018 15:35:36.814 +0000 ERROR loader - win-service: Error running pre-flight-checks (_pclose returned 10).
12-10-2018 15:35:36.814 +0000 ERROR loader - win-service: Here is the output from running pre-flight-checks:
12-10-2018 15:35:36.814 +0000 ERROR loader - Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3
12-10-2018 15:35:36.814 +0000 ERROR loader -
12-10-2018 15:35:36.814 +0000 ERROR loader -  Checking critical directories... Done
12-10-2018 15:35:36.814 +0000 ERROR loader -  Checking indexes...
12-10-2018 15:35:36.814 +0000 ERROR loader - Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
12-10-2018 15:35:36.814 +0000 ERROR loader - <<<<< EOF (pre-flight-checks)

any help is greatly appreciated

Willsy

Re: Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3"

prakash007 — Mon, 10 Dec 2018 19:03:42 GMT

@Willsy: It looks like your indexes are not consistent across all instances(including your search-head), run a btool on your search-head for indexes.conf, and also look for any errors on the Search-head _internal logs.
Compare other search-head configs with this search-head.

##check any invalid key-stanzas
$SPLUNK_HOME/bin/splunk cmd btool check 
$SPLUNK_HOME/bin/splunk  cmd btool indexes list --debug

Re: Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3"

dkeck — Mon, 10 Dec 2018 19:12:29 GMT

Hi,

could be a permission issue, does the account running splunk have access to D:\Splunk\cisco\db?

Re: Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3"

willsy — Tue, 11 Dec 2018 09:31:18 GMT

This is what i recieved from btool, it doesnt look as though anything is wrong.

C:\Program Files\Splunk\etc\system\local\indexes.conf [cisco]
C:\Program Files\Splunk\etc\system\default\indexes.conf archiver.enableDataArchive = false
C:\Program Files\Splunk\etc\system\default\indexes.conf archiver.maxDataArchiveRetentionPeriod = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf assureUTF8 = false
C:\Program Files\Splunk\etc\system\default\indexes.conf bucketRebuildMemoryHint = auto
C:\Program Files\Splunk\etc\system\local\indexes.conf coldPath = D:\Splunk\cisco\colddb
C:\Program Files\Splunk\etc\system\default\indexes.conf coldPath.maxDataSizeMB = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf coldToFrozenDir =
C:\Program Files\Splunk\etc\system\default\indexes.conf coldToFrozenScript =
C:\Program Files\Splunk\etc\system\default\indexes.conf compressRawdata = true
C:\Program Files\Splunk\etc\system\default\indexes.conf datatype = event
C:\Program Files\Splunk\etc\system\default\indexes.conf defaultDatabase = main
C:\Program Files\Splunk\etc\system\default\indexes.conf enableDataIntegrityControl = false
C:\Program Files\Splunk\etc\system\default\indexes.conf enableOnlineBucketRepair = true
C:\Program Files\Splunk\etc\system\default\indexes.conf enableRealtimeSearch = true
C:\Program Files\Splunk\etc\system\default\indexes.conf enableTsidxReduction = false
C:\Program Files\Splunk\etc\system\local\indexes.conf frozenTimePeriodInSecs = 2592000
C:\Program Files\Splunk\etc\system\local\indexes.conf homePath = D:\Splunk\cisco\db
C:\Program Files\Splunk\etc\system\default\indexes.conf homePath.maxDataSizeMB = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf hotBucketTimeRefreshInterval = 10
C:\Program Files\Splunk\etc\system\default\indexes.conf indexThreads = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf journalCompression = gzip
C:\Program Files\Splunk\etc\system\default\indexes.conf maxBloomBackfillBucketAge = 30d
C:\Program Files\Splunk\etc\system\default\indexes.conf maxBucketSizeCacheEntries = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf maxConcurrentOptimizes = 6
C:\Program Files\Splunk\etc\system\local\indexes.conf maxDataSize = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf maxGlobalDataSizeMB = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf maxHotBuckets = 3
C:\Program Files\Splunk\etc\system\default\indexes.conf maxHotIdleSecs = 0
C:\Program Files\Splunk\etc\system\local\indexes.conf maxHotSpanSecs = 432000
C:\Program Files\Splunk\etc\system\default\indexes.conf maxMemMB = 5
C:\Program Files\Splunk\etc\system\default\indexes.conf maxMetaEntries = 1000000
C:\Program Files\Splunk\etc\system\default\indexes.conf maxRunningProcessGroups = 8
C:\Program Files\Splunk\etc\system\default\indexes.conf maxRunningProcessGroupsLowPriority = 1
C:\Program Files\Splunk\etc\system\default\indexes.conf maxTimeUnreplicatedNoAcks = 300
C:\Program Files\Splunk\etc\system\default\indexes.conf maxTimeUnreplicatedWithAcks = 60
C:\Program Files\Splunk\etc\system\default\indexes.conf maxTotalDataSizeMB = 500000
C:\Program Files\Splunk\etc\system\default\indexes.conf maxWarmDBCount = 300
C:\Program Files\Splunk\etc\system\default\indexes.conf memPoolMB = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf minHotIdleSecsBeforeForceRoll = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf minRawFileSyncSecs = disable
C:\Program Files\Splunk\etc\system\default\indexes.conf minStreamGroupQueueSize = 2000
C:\Program Files\Splunk\etc\system\default\indexes.conf partialServiceMetaPeriod = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf processTrackerServiceInterval = 1
C:\Program Files\Splunk\etc\system\default\indexes.conf quarantineFutureSecs = 2592000
C:\Program Files\Splunk\etc\system\default\indexes.conf quarantinePastSecs = 77760000
C:\Program Files\Splunk\etc\system\default\indexes.conf rawChunkSizeBytes = 131072
C:\Program Files\Splunk\etc\system\local\indexes.conf repFactor = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf rotatePeriodInSecs = 60
C:\Program Files\Splunk\etc\system\default\indexes.conf rtRouterQueueSize = 10000
C:\Program Files\Splunk\etc\system\default\indexes.conf rtRouterThreads = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf selfStorageThreads = 2
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceInactiveIndexesPeriod = 60
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceMetaPeriod = 25
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceOnlyAsNeeded = true
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceSubtaskTimingPeriod = 30
C:\Program Files\Splunk\etc\system\default\indexes.conf splitByIndexKeys =
C:\Program Files\Splunk\etc\system\default\indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000
C:\Program Files\Splunk\etc\system\default\indexes.conf suppressBannerList =
C:\Program Files\Splunk\etc\system\default\indexes.conf suspendHotRollByDeleteQuery = false
C:\Program Files\Splunk\etc\system\default\indexes.conf sync = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf syncMeta = true
C:\Program Files\Splunk\etc\system\local\indexes.conf thawedPath = D:\Splunk\cisco\thaweddb

Re: Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3"

willsy — Tue, 11 Dec 2018 11:49:21 GMT

@prakash007 i have ran btools with the commands you have given, there was no issues, the indexes.conf for cisco were correct with no errors, i have tried to attach the output for my cisco stanzas and file path but it seemed ok.

@dkeck i have also tried your comment, i deleted the original cisco db D:\Splunk\cisco\db (it didnt have anything of value in as its new and in test) but to prove access and permissions i restarted my cluster and the D:\Splunk\cisco\db was there again. i then viewed permissions on the files and i have system properties so thats all good.

to you both, i am not wholly sure where to go from now on, unless i delete the cisco app as a whole and potentially start again. thoughts? or is there anything else i could try?

many thanks in advance

Re: Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3"

prakash007 — Tue, 11 Dec 2018 15:28:19 GMT

@willsy: so, you don't see any output when you run Splunk cmd btool check..??
It looks like the output you posted is from splunk cmd btool indexes list --debug.

what version of splunk are you on..??

Check your SPLUNK_DB environment variable, look at this splunk answer if it helps...
https://answers.splunk.com/answers/94428/error-warning-cannot-create-new-path-for-index-when-starting-splunk.html