Installation

I upgraded to 4.1.4 but now I get an error on startup - could not validate indexes - what does this mean?

mctester
Communicator

My Splunk instance was working fine before I upgraded to 4.1.4, and now I'm seeing this message. My earlier question on the subject was answered, but I haven't made any changes to my indexes recently, why is this happening now?

Tags (3)
1 Solution

Mick
Splunk Employee
Splunk Employee

It's the same problem as answered on your other question - one of the indexes.conf files within your instance is referencing an index that Splunk doesn't know exists. In this case, it's the fault of a product change, which removed an old legacy index called [splunklogger].

In the pre-4.1.4 days, this index was established in $SPLUNK_HOME/etc/system/default/indexes.conf, but it has now been removed. If your retirement policy referenced this index in any way, to limit the size it could grow to for example, then you just have to remove that setting from your $SPLUNK_HOME/etc/system/local/indexes.conf

View solution in original post

hexx
Splunk Employee
Splunk Employee

During the first start up after the upgrade, Splunk tries to validate the existence of all declared indexes in the indexes.conf found in your configuration directories.

Even if an index is disabled with "disabled = true" or by being present in the indexes.conf of a disabled app, Splunk will freak out if it cannot locate it's "homePath" (whether it's the default one in $SPLUNK_DB or a custom one specified in indexes.conf).

The quickest way to troubleshoot this problem is to compare the output of two commands.

The first one displays all existing index configuration stanzas across all indexes.conf :

# find $SPLUNK_HOME/etc/ -name indexes.conf | xargs grep ^\\[

The second command will list the directories present in $SPLUNK_DB ($SPLUNK_HOME/var/lib/splunk/ by default), which is the default "homePath" for each index if no other value for that parameter is specified.

# ls -l $SPLUNK_DB

or

# ls -l $SPLUNK_HOME/var/lib/splunk/

If the first command shows an index that is not present in $SPLUNK_DB and that index doesn't have a setting for "homePath" that points to an existing directory, you should consider disabling that index stanza in that configuration file by preceding it with a "#".

Typical culprits are "splunklogger" (which has been deprecated and is often present but disabled) and "os" (which is often present in disabled apps like "unix" or "nixLF").

This is particularly effective on a forwarder where there shouldn't be any indexes configured other than what can be found in $SPLUNK_DB.

Here's a typical scenario :

1) Which index directories exist?

# ls -l $SPLUNK_DB

total 52
drwx--x--x 5 root root 4096 Jun 25 16:29 audit
drwx--x--x 2 root root 4096 Jun 25 10:14 authDb
drwx--x--x 5 root root 4096 Jun 25 16:29 blockSignature
drwx--x--x 5 root root 4096 Jun 25 16:29 defaultdb
drwx--x--x 6 root root 4096 Jun 25 16:29 fishbucket
drwx--x--x 2 root root 4096 Jun 25 16:29 hashDb
drwx--x--x 5 root root 4096 Jun 25 16:29 historydb
drwx--x--x 5 root root 4096 Jun 25 16:29 _internaldb
drwx------ 2 root root 4096 Jun 25 16:29 persistentstorage
drwx------ 2 root root 4096 Jun 25 16:27 queues
drwx--x--x 5 root root 4096 Jun 25 16:29 sample
drwx--x--x 5 root root 4096 Jun 25 16:29 splunkloggerdb
drwx--x--x 5 root root 4096 Jun 25 16:29 summarydb

2) Which indexes are declared in the configuration files?

# find $SPLUNK_HOME/etc/ -name indexes.conf | xargs grep ^\\[

etc/system/default/indexes.conf:[main]
etc/system/default/indexes.conf:[history]
etc/system/default/indexes.conf:[summary]
etc/system/default/indexes.conf:[_internal]
etc/system/default/indexes.conf:[_audit]
etc/system/default/indexes.conf:[_thefishbucket]
etc/system/default/indexes.conf:[_blocksignature]
etc/system/default/indexes.conf:[splunklogger]
etc/apps/sample_app/default/indexes.conf:[sample]
etc/apps/nixLF/default/indexes.conf:[main]
etc/apps/nixLF/default/indexes.conf:[history]
etc/apps/nixLF/default/indexes.conf:[summary]
etc/apps/nixLF/default/indexes.conf:[_internal]
etc/apps/nixLF/default/indexes.conf:[_audit]
etc/apps/nixLF/default/indexes.conf:[_blocksignature]
etc/apps/nixLF/default/indexes.conf:[splunklogger]
etc/apps/unix/default/indexes.conf:[os]
etc/apps/SplunkLightForwarder/default/indexes.conf:[main]
etc/apps/SplunkLightForwarder/default/indexes.conf:[history]
etc/apps/SplunkLightForwarder/default/indexes.conf:[summary]
etc/apps/SplunkLightForwarder/default/indexes.conf:[_internal]
etc/apps/SplunkLightForwarder/default/indexes.conf:[_audit]
etc/apps/SplunkLightForwarder/default/indexes.conf:[_blocksignature]
etc/apps/SplunkLightForwarder/default/indexes.conf:[splunklogger]

This revealed the following culprits :

etc/apps/unix/default/indexes.conf:[os]
etc/system/default/indexes.conf:[splunklogger]
etc/apps/nixLF/default/indexes.conf:[splunklogger]
etc/apps/SplunkLightForwarder/default/indexes.conf:[splunklogger]

The "splunklogger" index was disabled everywhere it was declared with "disabled = true", and the "os" index was not in use as the unix app was disabled. Once these stanzas and all of their associated parameters were commented out, the upgrade worked!

Mick
Splunk Employee
Splunk Employee

It's the same problem as answered on your other question - one of the indexes.conf files within your instance is referencing an index that Splunk doesn't know exists. In this case, it's the fault of a product change, which removed an old legacy index called [splunklogger].

In the pre-4.1.4 days, this index was established in $SPLUNK_HOME/etc/system/default/indexes.conf, but it has now been removed. If your retirement policy referenced this index in any way, to limit the size it could grow to for example, then you just have to remove that setting from your $SPLUNK_HOME/etc/system/local/indexes.conf

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...