Hi Experts,
I currently suspect that one of the existing Splunk DB Connect 2 Operations job keep failing to retrieve the data due to the errors below. This is causing data lagging for 10 hours.
Anyone have a solution for this?
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Traceback (most recent call last):
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" File "C:\Program Files\Splunk\Python-2.7\Lib\logging\handlers.py", line 77, in emit
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" self.doRollover()
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" File "C:\Program Files\Splunk\Python-2.7\Lib\logging\handlers.py", line 142, in doRollover
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" os.rename(self.baseFilename, dfn)
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" WindowsError: [Error 32] The process cannot access the file because it is being used by another process
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Logged from file health.py, line 186
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Traceback (most recent call last):
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" File "C:\Program Files\Splunk\Python-2.7\Lib\logging\handlers.py", line 77, in emit
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" self.doRollover()
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" File "C:\Program Files\Splunk\Python-2.7\Lib\logging\handlers.py", line 142, in doRollover
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" os.rename(self.baseFilename, dfn)
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" WindowsError: [Error 32] The process cannot access the file because it is being used by another process
09-15-2015 12:00:46.469 +0800 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Logged from file health.py, line 193
This behavior was first noted in 2.3.0 and was subsequently resolved by DBX-3193 in version 2.3.1.
Please upgrade your DB Connect instance to 2.3.1 and let us know if you experience any further issues with this.
Thank you!
I'm not seeing any data lag, but my DB queries are only retrieving relatively small amounts of data. However, I've been seeing this error in my splunkd.log since Monday. I also noticed that since this error appeared, no updates have been made to the py_health.log file. I cleared down the py_health.log file and the messages have stopped appearing in the splunkd.log.
It seems this is a bug with DBConnect when the py_health.log hits 10,000KB. We noticed the same thing and stopping splunkd, renaming the file and starting splunkd solved the issue.
https://answers.splunk.com/answers/319066/why-is-the-splunk-db-connect-205-health-dashboard.html