Important: don't run the commands below if you aren't sure what they do. You could end up changing owner:group permissions on your entire system which is a pain in the arse.
Without much info to go on...it sounds like you might have restarted splunk as the wrong user. Are you using Linux? I am and I've done this before. On my set-up I run Splunk as the user 'Splunk'. All the files & folders should be owned by this user.
I found out a few hours after IT restarted Splunk as 'root' user that something was wrong. I restarted via the command line and dictated which user (Splunk) it should run under:
sudo -H -u splunk /$splunk_home_directory$/bin/splunk restart
This didn't solve the issue completely because, after IT restarted Splunk0 as 'root', newly indexed data and other files were now owned by 'root'. The symptom was that after I restarted Splunk as user 'splunk', I could not see anything indexed while SPlunk was running under 'root' user. My data only showed events from the day before back.
To fix, I stopped Splunk and changed owner:group on ever single file and directory in the splunk home directory:
From the parent directory of the splunk home directory:
sudo chown splunk:splunk -R splunk/
Then I restarted again:
sudo -H -u splunk /$splunk_home_directory$/bin/splunk restart
For some reason this didn't change some files so I had to do a search for any files in the Splunk directory that weren't owned by splunk user. I manually ran chown against these files, restarted splunk correctly, and voila. Back to normal.
... View more