Why am I unable to delete indexes from the Splunk Web? Why do I have to restart Splunk when I create a new Index from Splunk Web?

[dshakespeare_sp]

Customer reported several issue with Index Management using the Splunk Web:
- Unable to create new Indexes from Settings > Indexes > New Index (Splunk Web reports that a restart is required)
- Unable to create new Indexes from Data Inputs > Files & Directories > New (Splunk Web reports that a restart is required)
- Unable to delete Indexes from Settings > Indexes > Delete (Splunk Web will not accept any input apart from "Cancel")

[dshakespeare_sp · Accepted solution]

On investigation TWO issues were found.

• The "need a restart" type messages seen in the GUI suggested that the indexes.conf on disk had been manually changed while Splunk was running and Splunk had not been restarted so that the running version in memory matched the disk version.

This issue was supported by the following error in splunkd.log:

"unable to reload indexer after adding: reason='reload is not safe since a path has been deleted or modified for an index, or an index has been disabled. You must restart the Splunk Server, for your changes to take effect.'. Restart required." messages seen in splunkd.log.

• splunkd.log also reported the error

  ERROR IndexConfig - stanza=osstats Required parameter=homePath not configured

Using "splunk btool indexes list --debug to locate the osstats stanza it was observed that in $SPLUNK_HOME/etc/system/local/inputs.conf, the following was declared

[osstats]
disabled = 1

Even though the stanza is disabled Splunk needs a homePATH coldPATH and thawedPATH to tell Splunk where the data files are. Without these stanzas Splunk sees the indexes.conf as invalid and may have problem configuring changes.

The stanza was changed to read

[osstats]
coldPath = $SPLUNK_DB/osstats/colddb
homePath = $SPLUNK_DB/osstats/db
thawedPath = $SPLUNK_DB/osstats/thaweddb
disabled = 1 

On restarting Splunk, the delete index functionality was returned to normal operation