Getting Data In

Why am I unable to delete indexes from the Splunk Web? Why do I have to restart Splunk when I create a new Index from Splunk Web?

dshakespeare_sp
Splunk Employee
Splunk Employee

Customer reported several issue with Index Management using the Splunk Web:
- Unable to create new Indexes from Settings > Indexes > New Index (Splunk Web reports that a restart is required)
- Unable to create new Indexes from Data Inputs > Files & Directories > New (Splunk Web reports that a restart is required)
- Unable to delete Indexes from Settings > Indexes > Delete (Splunk Web will not accept any input apart from "Cancel")

1 Solution

dshakespeare_sp
Splunk Employee
Splunk Employee

On investigation TWO issues were found.

  • The "need a restart" type messages seen in the GUI suggested that the indexes.conf on disk had been manually changed while Splunk was running and Splunk had not been restarted so that the running version in memory matched the disk version.

This issue was supported by the following error in splunkd.log:

"unable to reload indexer after adding: reason='reload is not safe since a path has been deleted or modified for an index, or an index has been disabled. You must restart the Splunk Server, for your changes to take effect.'. Restart required." messages seen in splunkd.log.
  • splunkd.log also reported the error

    ERROR IndexConfig - stanza=osstats Required parameter=homePath not configured

Using "splunk btool indexes list --debug to locate the osstats stanza it was observed that in $SPLUNK_HOME/etc/system/local/inputs.conf, the following was declared

[osstats]
disabled = 1

Even though the stanza is disabled Splunk needs a homePATH coldPATH and thawedPATH to tell Splunk where the data files are. Without these stanzas Splunk sees the indexes.conf as invalid and may have problem configuring changes.

The stanza was changed to read

[osstats]
coldPath = $SPLUNK_DB/osstats/colddb
homePath = $SPLUNK_DB/osstats/db
thawedPath = $SPLUNK_DB/osstats/thaweddb
disabled = 1 

On restarting Splunk, the delete index functionality was returned to normal operation

View solution in original post

dshakespeare_sp
Splunk Employee
Splunk Employee

On investigation TWO issues were found.

  • The "need a restart" type messages seen in the GUI suggested that the indexes.conf on disk had been manually changed while Splunk was running and Splunk had not been restarted so that the running version in memory matched the disk version.

This issue was supported by the following error in splunkd.log:

"unable to reload indexer after adding: reason='reload is not safe since a path has been deleted or modified for an index, or an index has been disabled. You must restart the Splunk Server, for your changes to take effect.'. Restart required." messages seen in splunkd.log.
  • splunkd.log also reported the error

    ERROR IndexConfig - stanza=osstats Required parameter=homePath not configured

Using "splunk btool indexes list --debug to locate the osstats stanza it was observed that in $SPLUNK_HOME/etc/system/local/inputs.conf, the following was declared

[osstats]
disabled = 1

Even though the stanza is disabled Splunk needs a homePATH coldPATH and thawedPATH to tell Splunk where the data files are. Without these stanzas Splunk sees the indexes.conf as invalid and may have problem configuring changes.

The stanza was changed to read

[osstats]
coldPath = $SPLUNK_DB/osstats/colddb
homePath = $SPLUNK_DB/osstats/db
thawedPath = $SPLUNK_DB/osstats/thaweddb
disabled = 1 

On restarting Splunk, the delete index functionality was returned to normal operation

Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...