Why is the Search and Reporting default overwritten during start up?

[sarahafrin]

The default folder under SPLUNK_HOME/etc/apps/search has been overwritten and all my changes are now in a default.old./ folder. Now, my Search and Reporting app is invisible. This has caused an outage for all settings also. I can only see apps.conf in this new default folder which has the following contents:
[install]
install_source_checksum =

This new default folder is not even owned by the unix group Splunk but by the unix group 'user'.

If i try to delete this new default folder, rename default.old. to default and restart Splunk daemon, it does not work. The default gets overwritten again with the same problem.

Can anyone help in understanding what might be causing this?

[woodcock]

The documentation, training, and file headers are quite clear. You should never, ever, EVER modify ANYTHING inside of Splunk's default directories. If you do, you are breaking your install and ensuring upgrade problems. Create a local (in this case, SPLUNK_HOME/etc/apps/search/local/ and put your changes there. To be fair, not all files have warnings (and each should) but, for example, the commands.conf file in $SPLUNK_HOME/etc/apps/search/default/ starts with these lines:

#   Version 7.1.1
# DO NOT EDIT THIS FILE!
# Please make all changes to files in $SPLUNK_HOME/etc/system/local.
# To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/system/default
# into ../local and edit there.

Other files say it like this:

# Version 7.0.3
# DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local
# (See "Configuration file precedence" in the web documentation).
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.
# This file contains possible attributes and values you can use to
# configure inputs, distributed inputs and file system monitoring.

[gjanders]

Do you have this server connecting to a deployment server? And is the deployment server sending out a search application?

Another possibility is a search head cluster pushing out the search application, however a search head cluster will not push out default applications unless a particular switch is used...(note that when a search head restarts it would re-download the current config from the deployer in this scenario).