I currently have four instances of Splunk, which are synchronised on a daily basis (the first instance pushes updates to the rest).
A couple of days ago I noticed that the regex for two extracted fields have changed somehow so the searches wouldn't work anymore. When I altered the regex for them, everything worked fine until the next day - I realised that the changes have completely reverted after splunk service restarted. I also found these error messages when I ran "./splunk btool check --debug " (there are more of them, for all splunk users):
No spec file for: /opt/splunk/etc/apps/user-prefs/default/user-
No spec file for: /opt/splunk/etc/system/default/conf.conf
No spec file for: /opt/splunk/etc/system/default/datatypesbnf.c
No spec file for: /opt/splunk/etc/system/default/default-mode.c
No spec file for: /opt/splunk/etc/system/default/prefs.conf
Not sure if that's the reason for why changes are not saved or if it's just a separate issue, but I would still like to fix these.
Can anyone help, please?
What are spec files and could I possibly recover them if they were lost maybe, etc?
I suspect that your "synchronization" is broken. What do you mean by that?
The files that you show are basic configuration files that your splunk instance needs. In particular the files in the /opt/splunk/etc/system/default directory are important and should not be changed.
If you are attempting to replicate changes between Splunk instances, you should leave the default directories alone.
As for how to recover, you could restore them from a backup if you have one. Or you could reinstall Splunk.
Depending on how badly the synchronization has damaged your Splunk instances, reinstallation might be the best route to a stable and correct Splunk.