Python scripts invoked by the Splunk Forwarder never exit

[alekksi]

Hi guys,

I'm using the Splunk Universal Forwarder (v5.0.3 - same as our Splunk enterprise; hopefully we are updating to 6.1 soon) to invoke python scripts which pull data from a specific source and output it over stdout to Splunk.

Currently the scripts seem to never end. They sit there in perpetuity without ever finishing. Below is the ps -ef of the running script, fifteen minutes after execution.

root 12360 12332 0 14:47 ? 00:00:00 python /opt/splunkforwarder/etc/apps/PerfStat/bin/perfStatImpl.py

Script is relatively straightforward, but here is a pared down version, removing unnecessary code:

try:

    [...]

    command = Popen(my_object.getCommand(),shell=True, stdout=PIPE)

    my_parse = sarparser.sarParser(argv[1], sar_gen.getDate())

    my_parse.parse(command)

    my_parse.output()

except:

    [...]

    exit(2)

exit(0)

Does anyone have any ideas? I've had this process (non-)running for almost an hour now, yet nothing has been output to Splunk! I can tell the script has completed as the log files have fully completed.

[alekksi]

Resolved now -- forwarder had some dodgy config (I blame ITSec 🙂 and splunk was crapping out accepting the amount of data I was throwing at it. Fixing the config and reducing the amount of data at one time sorted this out.

[lguinn2]

Here are a couple of ideas:

1 - what happens if you simply execute the script from the command line?

cd $SPLUNK_HOME/etc/apps/yourapp/bin
./yourscript.py

2 - what happens if you ask Splunk to execute the script in its environment?

cd $SPLUNK_HOME/bin
./splunk cmd $SPLUNK_HOME/etc/apps/yourapp/bin/yourscript.py

My guess is that it will hang in one of those two scenarios as well, but this should help you debug.

[alekksi]

Turns out that Splunk is taking issue with the amount of data I'm feeding it at once. It processes ~7-10 thousand lines before it just gives up on it.

[alekksi]

Thanks for the ideas. Unfortunately both of them return the expected output. Even the process spawned by splunk for the script has the correct string which works if run. I've even rewritten the script slightly to use the .communicate() function which apparently reduces potential deadlocks.

Very confused.