Monitoring Splunk

Can you help me with the following error that showed up after restarting the server: "Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3"

willsy
Communicator

I restarted my server, and the Splunk web GUI didn't load up. My other servers and search heads load up, just not this particular search head. I know the issue is meant to be multiple indexes of the same thing, but I can't seem to see which one would be the problem child. This is the error i continually get every time i try, and either manually restart splunk services or restart the machine again.

12-10-2018 15:35:27.962 +0000 INFO  loader - win-service: Starting as a Windows service: will run various system checks first...
12-10-2018 15:35:27.962 +0000 INFO  loader - win-service: Splunk starting as a local administrator
12-10-2018 15:35:27.962 +0000 INFO  loader - Automatic migration of modular inputs
12-10-2018 15:35:36.814 +0000 ERROR loader - win-service: Error running pre-flight-checks (_pclose returned 10).
12-10-2018 15:35:36.814 +0000 ERROR loader - win-service: Here is the output from running pre-flight-checks:
12-10-2018 15:35:36.814 +0000 ERROR loader - Could not create path D:\Splunk\cisco\db appearing in indexes.conf: 3
12-10-2018 15:35:36.814 +0000 ERROR loader -
12-10-2018 15:35:36.814 +0000 ERROR loader -  Checking critical directories... Done
12-10-2018 15:35:36.814 +0000 ERROR loader -  Checking indexes...
12-10-2018 15:35:36.814 +0000 ERROR loader - Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue
12-10-2018 15:35:36.814 +0000 ERROR loader - <<<<< EOF (pre-flight-checks)

any help is greatly appreciated

Willsy

0 Karma

prakash007
Builder

@willsy: so, you don't see any output when you run Splunk cmd btool check..??
It looks like the output you posted is from splunk cmd btool indexes list --debug.

what version of splunk are you on..??

Check your SPLUNK_DB environment variable, look at this splunk answer if it helps...
https://answers.splunk.com/answers/94428/error-warning-cannot-create-new-path-for-index-when-startin...

0 Karma

willsy
Communicator

@prakash007 i have ran btools with the commands you have given, there was no issues, the indexes.conf for cisco were correct with no errors, i have tried to attach the output for my cisco stanzas and file path but it seemed ok.

@dkeck i have also tried your comment, i deleted the original cisco db D:\Splunk\cisco\db (it didnt have anything of value in as its new and in test) but to prove access and permissions i restarted my cluster and the D:\Splunk\cisco\db was there again. i then viewed permissions on the files and i have system properties so thats all good.

to you both, i am not wholly sure where to go from now on, unless i delete the cisco app as a whole and potentially start again. thoughts? or is there anything else i could try?

many thanks in advance

0 Karma

dkeck
Influencer

Hi,

could be a permission issue, does the account running splunk have access to D:\Splunk\cisco\db?

0 Karma

prakash007
Builder

@Willsy: It looks like your indexes are not consistent across all instances(including your search-head), run a btool on your search-head for indexes.conf, and also look for any errors on the Search-head _internal logs.
Compare other search-head configs with this search-head.

##check any invalid key-stanzas
$SPLUNK_HOME/bin/splunk cmd btool check 
$SPLUNK_HOME/bin/splunk  cmd btool indexes list --debug
0 Karma

willsy
Communicator

This is what i recieved from btool, it doesnt look as though anything is wrong.

C:\Program Files\Splunk\etc\system\local\indexes.conf [cisco]
C:\Program Files\Splunk\etc\system\default\indexes.conf archiver.enableDataArchive = false
C:\Program Files\Splunk\etc\system\default\indexes.conf archiver.maxDataArchiveRetentionPeriod = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf assureUTF8 = false
C:\Program Files\Splunk\etc\system\default\indexes.conf bucketRebuildMemoryHint = auto
C:\Program Files\Splunk\etc\system\local\indexes.conf coldPath = D:\Splunk\cisco\colddb
C:\Program Files\Splunk\etc\system\default\indexes.conf coldPath.maxDataSizeMB = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf coldToFrozenDir =
C:\Program Files\Splunk\etc\system\default\indexes.conf coldToFrozenScript =
C:\Program Files\Splunk\etc\system\default\indexes.conf compressRawdata = true
C:\Program Files\Splunk\etc\system\default\indexes.conf datatype = event
C:\Program Files\Splunk\etc\system\default\indexes.conf defaultDatabase = main
C:\Program Files\Splunk\etc\system\default\indexes.conf enableDataIntegrityControl = false
C:\Program Files\Splunk\etc\system\default\indexes.conf enableOnlineBucketRepair = true
C:\Program Files\Splunk\etc\system\default\indexes.conf enableRealtimeSearch = true
C:\Program Files\Splunk\etc\system\default\indexes.conf enableTsidxReduction = false
C:\Program Files\Splunk\etc\system\local\indexes.conf frozenTimePeriodInSecs = 2592000
C:\Program Files\Splunk\etc\system\local\indexes.conf homePath = D:\Splunk\cisco\db
C:\Program Files\Splunk\etc\system\default\indexes.conf homePath.maxDataSizeMB = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf hotBucketTimeRefreshInterval = 10
C:\Program Files\Splunk\etc\system\default\indexes.conf indexThreads = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf journalCompression = gzip
C:\Program Files\Splunk\etc\system\default\indexes.conf maxBloomBackfillBucketAge = 30d
C:\Program Files\Splunk\etc\system\default\indexes.conf maxBucketSizeCacheEntries = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf maxConcurrentOptimizes = 6
C:\Program Files\Splunk\etc\system\local\indexes.conf maxDataSize = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf maxGlobalDataSizeMB = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf maxHotBuckets = 3
C:\Program Files\Splunk\etc\system\default\indexes.conf maxHotIdleSecs = 0
C:\Program Files\Splunk\etc\system\local\indexes.conf maxHotSpanSecs = 432000
C:\Program Files\Splunk\etc\system\default\indexes.conf maxMemMB = 5
C:\Program Files\Splunk\etc\system\default\indexes.conf maxMetaEntries = 1000000
C:\Program Files\Splunk\etc\system\default\indexes.conf maxRunningProcessGroups = 8
C:\Program Files\Splunk\etc\system\default\indexes.conf maxRunningProcessGroupsLowPriority = 1
C:\Program Files\Splunk\etc\system\default\indexes.conf maxTimeUnreplicatedNoAcks = 300
C:\Program Files\Splunk\etc\system\default\indexes.conf maxTimeUnreplicatedWithAcks = 60
C:\Program Files\Splunk\etc\system\default\indexes.conf maxTotalDataSizeMB = 500000
C:\Program Files\Splunk\etc\system\default\indexes.conf maxWarmDBCount = 300
C:\Program Files\Splunk\etc\system\default\indexes.conf memPoolMB = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf minHotIdleSecsBeforeForceRoll = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf minRawFileSyncSecs = disable
C:\Program Files\Splunk\etc\system\default\indexes.conf minStreamGroupQueueSize = 2000
C:\Program Files\Splunk\etc\system\default\indexes.conf partialServiceMetaPeriod = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf processTrackerServiceInterval = 1
C:\Program Files\Splunk\etc\system\default\indexes.conf quarantineFutureSecs = 2592000
C:\Program Files\Splunk\etc\system\default\indexes.conf quarantinePastSecs = 77760000
C:\Program Files\Splunk\etc\system\default\indexes.conf rawChunkSizeBytes = 131072
C:\Program Files\Splunk\etc\system\local\indexes.conf repFactor = auto
C:\Program Files\Splunk\etc\system\default\indexes.conf rotatePeriodInSecs = 60
C:\Program Files\Splunk\etc\system\default\indexes.conf rtRouterQueueSize = 10000
C:\Program Files\Splunk\etc\system\default\indexes.conf rtRouterThreads = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf selfStorageThreads = 2
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceInactiveIndexesPeriod = 60
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceMetaPeriod = 25
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceOnlyAsNeeded = true
C:\Program Files\Splunk\etc\system\default\indexes.conf serviceSubtaskTimingPeriod = 30
C:\Program Files\Splunk\etc\system\default\indexes.conf splitByIndexKeys =
C:\Program Files\Splunk\etc\system\default\indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000
C:\Program Files\Splunk\etc\system\default\indexes.conf suppressBannerList =
C:\Program Files\Splunk\etc\system\default\indexes.conf suspendHotRollByDeleteQuery = false
C:\Program Files\Splunk\etc\system\default\indexes.conf sync = 0
C:\Program Files\Splunk\etc\system\default\indexes.conf syncMeta = true
C:\Program Files\Splunk\etc\system\local\indexes.conf thawedPath = D:\Splunk\cisco\thaweddb

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...