Configuration bundle not working

[sympatiko]

Hi,

Good day! I'm getting the following error. I already successfully pushed the /_cluster/local/fortigate config from master to my 2 indexers. I already restarted the indexers. The repFactor=auto is already set.

Search peer indexer1 has the following message: received event for unconfigured/disabled/deleted index='fortigate' with source='source::/var/log/fortigate/fortigate.log' host='host::proxy' sourcetype='sourcetype::fortigate'

Search peer indexer2 has the following message: received event for unconfigured/disabled/deleted index='fortigate' with source='source::/var/log/fortigate/fortigate.log' host='host::proxy' sourcetype='sourcetype::fortigate' (1 missing total)

Thanks,

[martin_mueller]

I see... so you have a master-apps/_cluster/local/fortigate/indexes.conf?

That fortigate directory in between shouldn't be there, either master-apps/_cluster/local/indexes.conf or master-apps/fortigate/local/indexes.conf.

The entire indexes.conf looks a bit messy as well, why are you adding a fortigate subdirectory into the path of Splunk-internal indexes? That confuses me a little. Also, when adding that to splunklogger, you forgot a few slashes.

[martin_mueller]

I don't even begin to comprehend why you've added a fortigate subdirectory to the path of all your indexes, including the summary index you posted the error message about.

[sympatiko]

Hi, Why I can't search any data? But I saw on the indexes tab that it is updating.

[sympatiko]

So I shouldn't do it that way?

[martin_mueller]

No, there can be any number of indexes.conf file.

However, Splunk isn't going to care about anything written in .../local/whatever/*.conf because that's not where configuration files go. The structure is (app-name)/(default-or-local)/.conf. Whether you use an app or the special _cluster app does not matter from a functional point of view, but you should group configuration that belongs together into the same place.

[sympatiko]

ohh,, thanks for the info. Can you try to look again to my indexes.conf? I'm having the ff error:

11-23-2014 03:03:03.306 +0800 ERROR IndexConfig - idx=summary Path coldPath='/opt/splunk/var/lib/splunk/fortigate/summarydb/colddb' (realpath '/opt/splunk/var/lib/splunk/fortigate/summarydb/colddb') is inside volume=fortigate (path='/opt/splunk/var/lib/splunk/fortigate', realpath='/opt/splunk/var/lib/splunk/fortigate'), but does not reference that volume. Space used by coldPath will not be volume-mananged. Config error?

[sympatiko]

That's quite messy here upon posting 😃 . So meaning to say there can be only one indexes.conf that can be define? Is the props.conf and transform.conf also required to be in master-apps/_cluster/local/ or master-apps/fortigate/local/ ?

[sympatiko]

Hi martin,

Please see details. I didn't change any thing from props.conf and transform.conf

$:/opt/splunk/etc/master-apps# ls
_cluster
$:/opt/splunk/etc/master-apps# cd _cluster/
$:/opt/splunk/etc/master-apps/_cluster# ls
default local
$:/opt/splunk/etc/master-apps/_cluster#

$:/opt/splunk/etc/master-apps/_cluster/local/fortigate# pwd
/opt/splunk/etc/master-apps/_cluster/local/fortigate
$:/opt/splunk/etc/master-apps/_cluster/local/fortigate# ls
indexes.conf props.conf transforms.conf
$:/opt/splunk/etc/master-apps/_cluster/local/fortigate#

indexes.conf

sync = 0
indexThreads = auto
memPoolMB = auto
defaultDatabase = main
blockSignatureDatabase = _blocksignature
enableRealtimeSearch = true
suppressBannerList =
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
bucketRebuildMemoryHint = auto
serviceOnlyAsNeeded = true
serviceSubtaskTimingPeriod = 30
maxBucketSizeCacheEntries = 0
processTrackerServiceInterval = 1
hotBucketTimeRefreshInterval = 10

maxDataSize = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
rotatePeriodInSecs = 60
coldToFrozenScript =
coldToFrozenDir =
compressRawdata = true
maxTotalDataSizeMB = 500000
maxMemMB = 5
maxConcurrentOptimizes = 6
blockSignSize = 0
maxHotSpanSecs = 7776000
maxHotIdleSecs = 0
maxHotBuckets = 3
quarantinePastSecs = 77760000
quarantineFutureSecs = 2592000
rawChunkSizeBytes = 131072
minRawFileSyncSecs = disable
assureUTF8 = false
serviceMetaPeriod = 25
partialServiceMetaPeriod = 0
throttleCheckPeriod = 15
syncMeta = true
maxMetaEntries = 1000000
maxBloomBackfillBucketAge = 30d
enableOnlineBucketRepair = true
maxTimeUnreplicatedWithAcks = 60
maxTimeUnreplicatedNoAcks = 300
minStreamGroupQueueSize = 2000
warmToColdScript=
tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0
streamingTargetTsidxSyncPeriodMsec = 5000

repFactor = auto

[volume:fortigate]
path = $SPLUNK_DB

[fortigate]
repFactor=auto
homePath = $SPLUNK_DB/fortigate/db
coldPath = $SPLUNK_DB/fortigate/colddb
thawedPath = $SPLUNK_DB/fortigate/thaweddb
tstatsHomePath = volume:fortigate/datamodel_summary
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume

[history]
homePath = $SPLUNK_DB/fortigate/historydb/db
coldPath = $SPLUNK_DB/fortigate/historydb/colddb
thawedPath = $SPLUNK_DB/fortigate/historydb/thaweddb
tstatsHomePath = volume:fortigate/historydb/datamodel_summary
maxDataSize = 10
frozenTimePeriodInSecs = 604800

[summary]
homePath = $SPLUNK_DB/fortigate/summarydb/db
coldPath = $SPLUNK_DB/fortigate/summarydb/colddb
thawedPath = $SPLUNK_DB/fortigate/summarydb/thaweddb
tstatsHomePath = volume:fortigate/summarydb/datamodel_summary

[_internal]
homePath = $SPLUNK_DB/fortigate/_internaldb/db
coldPath = $SPLUNK_DB/fortigate/_internaldb/colddb
thawedPath = $SPLUNK_DB/fortigate/_internaldb/thaweddb
tstatsHomePath = volume:fortigate/_internaldb/datamodel_summary
maxDataSize = 1000
maxHotSpanSecs = 432000
frozenTimePeriodInSecs = 2592000

[_audit]
homePath = $SPLUNK_DB/fortigate/audit/db
coldPath = $SPLUNK_DB/fortigate/audit/colddb
thawedPath = $SPLUNK_DB/fortigate/audit/thaweddb
tstatsHomePath = volume:fortigate/audit/datamodel_summary

[_thefishbucket]
homePath = $SPLUNK_DB/fortigate/fishbucket/db
coldPath = $SPLUNK_DB/fortigate/fishbucket/colddb
thawedPath = $SPLUNK_DB/fortigate/fishbucket/thaweddb
tstatsHomePath = volume:fortigate/fishbucket/datamodel_summary
maxDataSize = 500
frozenTimePeriodInSecs = 2419200

[_blocksignature]
homePath = $SPLUNK_DB/fortigate/blockSignature/db
coldPath = $SPLUNK_DB/fortigate/blockSignature/colddb
thawedPath = $SPLUNK_DB/fortigate/blockSignature/thaweddb
tstatsHomePath = volume:fortigateblockSignature/datamodel_summary
maxDataSize = 1000
frozenTimePeriodInSecs = 0
maxTotalDataSizeMB = 0

[splunklogger]
homePath = $SPLUNK_DB/fortigate/splunklogger/db
coldPath = $SPLUNK_DB/fortigatesplunklogger/colddb
thawedPath = $SPLUNK_DB/fortigatesplunklogger/thaweddb
disabled = true

[_introspection]
homePath = $SPLUNK_DB/fortigate/_introspection/db
coldPath = $SPLUNK_DB/fortigate/_introspection/colddb
thawedPath = $SPLUNK_DB/fortigate/_introspection/thaweddb
maxDataSize = 1024
frozenTimePeriodInSecs = 1209600
root@master:/opt/splunk/etc/master-apps/_cluster/local/fortigate#

Thanks,

[martin_mueller]

Do post the directory tree under master-apps and the content of the relevant config files.