while refreshing our Splunk installation we reinstalled the various TAs and Apps. Currently we are working on the "Splunk App for Windows Infrastructure" (version 5.0.1). This requires "Splunk Add-on for Microsoft Active Directory" (still at 1.0.0).
This leads to inconsistencies: The AD addon defines the various "old" indices (msad, perfmon, wineventlog etc.), but the new "App for Windows" does not depend on this structure anymore. Thus I wanted to gather all windows data in a single index named "windowsindex", which basically works by defining this index in all inputs.conf files inside the [default] stanza.
Now the complicated part: I can delete the "old" and now unused indices like "msad" successfully, no problem here. BUT: after Splunk restart this index is recreated again, probably because it is defined in the inputs.conf in the "/default"-tree of the AD addon. Otherwise editing the files in the /default-directories is kind of forbidden...
So how can I get rid of indices defined inside the /default-directory of an app or addon??
Alternative solution (for this case): Upgrade the AD addon to be in sync with the new structure of the Splunk Windows App (hint, hint).
Any ideas for a work around?
Thanks and best regards,
OK, answering my own question...
The analysis seems to be correct: if an index is defined in an indexes.conf file, which is contained in a .../default-directory, then this indexes will get recreated after a restart of Splunk! Even if this index was deleted completely. It will get the "deleted = true" flag, in the indexes.conf file residing in the .../local-directory.
Thus you have to comment out the stanza for the index to be deleted in the /default/indexes.conf and to add the contexts to the /local/indexes.conf, THEN the index will get deleted completly during restart and it will NOT be recreated. Weird logic... and an exeception to the rule of not editing .conf-files inside a /default-directory...
More observations (tested with the index "msad") after a few restarts: Index is still there and the indexes.conf in the /local-directory always shows this
deleted = true
disabled = true