When I went to the directory in question on the indexer and saw that there were a bunch of files all owned by 1010:1010. After a quick look at the /etc/passwd file on the indexer I noticed that there were no UIDs associated with the 1010 ID. I then went back to the search head and saw that indeed, splunk has an ID of 1010.
So, I see what the problem is but how do I address it? Who is supposed to own this directory? If I manually change the ownership of the directory and associated subdirectories, will Splunk override them with 1010:1010 again?
Splunk normally works fine when the splunk user (often 'splunk') owns everything under /opt/splunk.
Strange things (mainly permissions issues) can happen when a splunk instance that normally runs under a restricted account, such as 'splunk', has been started as 'root' and then after a while, is restarted under the correct user.
In the meantime quite a few files have been altered or created, and the 'splunk' account has no (or limitied) access.
You could try to change ownership, but you should of course find out first which user splunk should run as.