All Apps and Add-ons

Python script execution fails

arunpsz
New Member

Hello, I have an add-on which fetches emails from mail server. Due to O365 limitations, the add-on's python script required modification to support OAuth authentication.

 

Post modification, the script runs, however, it appears to be crashing once in a while randomly. After enabling the DEBUG for ExecProcessor, the following info is seen in the logs:

 

07-18-2023 15:29:34.771 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Got EOF from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py", uniqueId=20706
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Ran script: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py, took 149.201812652 seconds to run, 0 bytes read 0 events read, status=done, exit=0, utime_sec=52.214078, stime_sec=7.235928, max_rss_kb=331052, vm_minor=606863, sched_vol=152097, sched_invol=3187
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Destroying ExecedCommandPipe for "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py" id=20706
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - ExecProcessorSharedState::addToRunQueue() path='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' restartTimerIfNeeded=1
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - adding "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py" to runqueue
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Added to run queue
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - Running: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py on PipelineSet 0
07-18-2023 15:29:34.781 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - PipelineSet 0: Created new ExecedCommandPipe for "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py", uniqueId=20741
07-18-2023 15:29:58.340 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:30:58.337 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:31:58.339 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:32:58.339 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:33:58.340 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue
07-18-2023 15:34:58.342 +0530 DEBUG ExecProcessor [1181322 ExecProcessor] - cmd='/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py' Not added to run queue

 

I also noticed a few processes of script in interruptible sleep state since many days.

 

root@splunk:~# ps aux | grep splunk
root 303533 0.0 0.7 129636 121296 ? S Jul14 0:21 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 375980 0.0 0.4 98060 81632 ? S Jul14 0:18 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
....
root 1221826 0.0 0.0 3968 1336 ? S 18:28 0:00 /bin/sh -c /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 1221827 11.8 0.8 149052 140628 ? S 18:28 0:19 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
....
root 1536311 0.0 0.5 113036 96648 ? S Jul05 0:25 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 2278791 0.0 0.5 100092 91804 ? S Jul07 0:23 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 3247873 0.0 0.6 114140 106092 ? S Jul10 0:25 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py
root 3704217 0.0 0.8 148820 140472 ? S Jul12 0:19 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-mailclient/bin/mail.py

 

arunpsz_0-1689675762863.png

 

Any help to debug further/fix the issue is highly appreciated. Thank you.

 

Regards,

Arun

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...