We recently upgraded from 7.2.1 to 7.3.3 and from the _internal
logs I can see that these new warnings are showing up across my indexer cluster. What is it saying and how do I go about fixing this? I've noticed that now I have an indexer just randomly lock up once a week. Any insight would be appreciated!
01-14-2020 00:58:27.994 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=_introspection Path coldPath='/opt/splunk/var/lib/splunk/_introspection/colddb' (realpath '/mnt/local/hot/_introspection/colddb') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by coldPath will *not* be volume-mananged. Please check indexes.conf for configuration errors.
01-14-2020 00:58:27.995 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=_telemetry Path coldPath='/opt/splunk/var/lib/splunk/_telemetry/colddb' (realpath '/mnt/local/hot/_telemetry/colddb') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by coldPath will *not* be volume-mananged. Please check indexes.conf for configuration errors.
01-14-2020 00:58:28.008 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=firedalerts Path coldPath='/opt/splunk/var/lib/splunk/firedalerts/colddb' (realpath '/mnt/local/hot/firedalerts/colddb') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by coldPath will *not* be volume-mananged. Please check indexes.conf for configuration errors.
01-14-2020 00:58:28.042 +0000 WARN ProcessTracker - (child_17__Fsck) IndexConfig - idx=wineventlog Path homePath='/opt/splunk/var/lib/splunk/wineventlog/db' (realpath '/mnt/local/hot/wineventlog/db') is inside volume=primary (path='/mnt/local/hot', realpath='/mnt/local/hot'), but does not reference that volume. Space used by homePath will *not* be volume-mananged. Please check indexes.conf for configuration errors.
indexes.conf
# global settings
[default]
lastChanceIndex = lastchance
malformedEventIndex = malformedevent
[volume:primary]
path = /mnt/local/hot
maxVolumeDataSizeMB = 14000000
[volume:cold]
path = /mnt/local/cold
maxVolumeDataSizeMB = 58200000
[volume:_splunk_summaries]
path = /mnt/local/hot
maxVolumeDataSizeMB = 1000000
homePath = volume:primary/$_index_name/db
coldPath = volume:cold/$_index_name/colddb
thawedPath = /mnt/local/cold/$_index_name/thaweddb
homePath.maxDataSizeMB = 2000000
maxWarmDBCount = 250
maxDataSize = auto
enableDataIntegrityControl = true
frozenTimePeriodInSecs = 188697600
[main]
homePath = volume:primary/defaultdb/db
coldPath = volume:cold/defaultdb/colddb
coldToFrozenDir = /mnt/local/cold/frozen/defaultdb
thawedPath = /mnt/local/cold/defaultdb/thaweddb
maxDataSize = auto_high_volume
frozenTimePeriodInSecs = 31536000
...
...
...
Lines 18-25 of the indexes.conf file above look like they're missing a stanza heading. Are they meant to be part of the [default] stanza? If so, try moving them up to under line 4.