The messages below appear in splunkd.log from splunk_app_infrastructure. The splunk_app_infrastructure is running fine. Searching is also fine, only indexing lags so only old data is found.
Additionally, when I try to upgrade an app, I get "Oops" and a drawing of a horse and "500 Internal Server Error".
Here are the messages in splunkd.log - it seems to be an intense app so it prevents indexing (I think, but know not):
ERROR ExecProcessor - message from "python D:\Programs\Splunk\etc\apps\splunk_app_infrastructure\bin\em_entity_manager.py"
Prepended to all the lines below
Traceback (most recent call last):
File "D:\Programs\Splunk\Python-2.7\Lib\logging\handlers.py", line 77, in emit
self.doRollover()
File "D:\Programs\Splunk\Python-2.7\Lib\logging\handlers.py", line 142, in doRollover
os.rename(self.baseFilename, dfn)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process
Logged from file em_model_entity.py, line 77
The messages "The process cannot access the file because it is being used by another process" could be caused by straggling process that still has a file open and we've spawned another process that trying to roll the logs going by: "File "D:\Programs\Splunk\Python-2.7\Lib\logging\handlers.py", line 142, in doRollover os.rename(self.baseFilename, dfn)".
Try: Stop splunk, check with Task Manager that there are no processes from the splunk directories left (like those python ones) and restart splunk after.
More on "WindowsError: [Error 32] The process cannot access the file because it is being used by another process" Kindly check https://github.com/grangier/python-goose/issues/78