Hello, I have an add-on which fetches emails from mail server. Due to O365 limitations, the add-on's python script required modification to support OAuth authentication.
Post modification, the script runs, however, it appears to be crashing once in a while randomly. After enabling the DEBUG for ExecProcessor, the following info is seen in the logs:
07-18-2023 15:29:34.771 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Got EOF from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py", uniqueId=20706
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Ran script: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py, took 149.201812652 seconds to run, 0 bytes read 0 events read, status=done, exit=0, utime_sec=52.214078, stime_sec=7.235928, max_rss_kb=331052, vm_minor=606863, sched_vol=152097, sched_invol=3187
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Destroying ExecedCommandPipe for "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py" id=20706
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - ExecProcessorSharedState::addToRunQueue() path='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' restartTimerIfNeeded=1
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - adding "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py" to runqueue
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Added to run queue
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - Running: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py on PipelineSet 0
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Created new ExecedCommandPipe for "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py", uniqueId=20741
07-18-2023 15:29:58.340 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:30:58.337 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:31:58.339 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:32:58.339 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:33:58.340 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:34:58.342 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
I also noticed a few processes of script in interruptible sleep state since many days.
root@splunk:~# ps aux | grep splunk
root 303533 0.0 0.7 129636 121296 ? S Jul14 0:21 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 375980 0.0 0.4 98060 81632 ? S Jul14 0:18 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
....
root 1221826 0.0 0.0 3968 1336 ? S 18:28 0:00 /bin/sh -c /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 1221827 11.8 0.8 149052 140628 ? S 18:28 0:19 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
....
root 1536311 0.0 0.5 113036 96648 ? S Jul05 0:25 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 2278791 0.0 0.5 100092 91804 ? S Jul07 0:23 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 3247873 0.0 0.6 114140 106092 ? S Jul10 0:25 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 3704217 0.0 0.8 148820 140472 ? S Jul12 0:19 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
Any help to debug further/fix the issue is highly appreciated. Thank you.
Regards,
Arun
