Customer reported several issue with Index Management using the Splunk Web:
- Unable to create new Indexes from Settings > Indexes > New Index (Splunk Web reports that a restart is required)
- Unable to create new Indexes from Data Inputs > Files & Directories > New (Splunk Web reports that a restart is required)
- Unable to delete Indexes from Settings > Indexes > Delete (Splunk Web will not accept any input apart from "Cancel")
The "need a restart" type messages seen in the GUI suggested that the indexes.conf on disk had been manually changed while Splunk was running and Splunk had not been restarted so that the running version in memory matched the disk version.
This issue was supported by the following error in splunkd.log:
"unable to reload indexer after adding: reason='reload is not safe since a path has been deleted or modified for an index, or an index has been disabled. You must restart the Splunk Server, for your changes to take effect.'. Restart required." messages seen in splunkd.log.
splunkd.log also reported the error
ERROR IndexConfig - stanza=osstats Required parameter=homePath not configured
Using "splunk btool indexes list --debug to locate the osstats stanza it was observed that in $SPLUNK_HOME/etc/system/local/inputs.conf, the following was declared
disabled = 1
Even though the stanza is disabled Splunk needs a homePATH coldPATH and thawedPATH to tell Splunk where the data files are. Without these stanzas Splunk sees the indexes.conf as invalid and may have problem configuring changes.