Modular Input Scripts Don't Die during Splunk restart

[alacercogitatus]

I have written two Modular Inputs for Splunk. Both exhibit the same behavior.

Steps to reproduce:

1. Issue "splunk restart"
2. ps -ef | grep python

Each Modular Input Data Input python script is orphaned after the restart, and when Splunk starts back up, it instantiates a new python process for each Data Input. Very quickly causes the box to become unresponsive, especially during dev work. I have not noticed this behavior on Windows or Mac OS X.

Linux: Ubuntu Linux #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Example:

ps -ef yields:
root 28238 28237 0 19:36 ? 00:00:01 python /opt/splunkbeta/etc/apps/GoogleApps/bin/googleapps.py

splunk stop

ps -ef yields:
root 28238 1 0 19:36 ? 00:00:01 python /opt/splunkbeta/etc/apps/GoogleApps/bin/googleapps.py

Am I missing something or is this a bug?

[kschon_splunk]

Another suggestion: The standard out and standard err of a scripted or modular input is piped back to Splunk, so when Splunk shuts down, the pipes are broken. So in Java, you can use:

boolean isSplunkRunning() {
    return !System.out.checkError();
}

This will return false the first time you write an event after Splunk shuts down. This is simpler than recording the PID and works on any OS.

[Damien_Dallimor]

If you use STDOUT. Many of my offerings allow you to ignore STDOUT (it performs poorly) and use HEC as the output "pipe" to Splunk.

[kschon_splunk]

Fair enough. A lot of folks will periodically write whitespace chars to std out to test the pipe.

BTW, the github link above gives me a 404. Does it still exist someplace?

[Damien_Dallimor]

Hyperlinking fixed , although it was showing the correct text. Hard to keep on top of answers from 2012 🙂

[kschon_splunk]

Thanks, I can see it now.

[igor]

Two questions come to mind: 1) Does "splunk restart" result in a clean shutdown? (You should see a line in splunkd.log that says "loader - All pipelines finished."). 2) Is it the case that the python script that splunkd spawns calls another python script? When splunkd shuts down, it sends SIGTERM to the process that it spawned; but if that process spawned another, that one won't be killed/signaled. That requires special handling.

[Damien_Dallimor · Accepted solution]

I also experienced this and have raised the issue.
In the meantime, you can look at how I implemented a workaround for my JMS Messaging Modular Input :
https://github.com/damiendallimore/SplunkModularInputsJavaFramework/tree/master/jms

Basically the mod input script (jms.py) writes a PID file that gets checked upon startup.
Also, the Java program that the jms.py script executes has some simple logic to check whether Splunk is still up , and if not , kills itself.

This ensures that for "splunk start|restart" there will only be 1 mod input process running , and for "splunk stop" there will be zero mod input processes running.

[alacercogitatus]

Yep, I do the same thing in My MIs. I check for grandparent pid on non-windows OS and if 1, exit. I wanted to see who else was seeing this, thanks!

[Damien_Dallimor]

With modular inputs you can specify in the scheme whether to run in single instance or multi instance mode. My JMS input mentioned above runs in single instance mode.

[csharp_splunk]

I've also considered implementing this in my scripted inputs. It's a good idea for anyone writing them, as the scripted input (and by extension, the modular input) is designed as much to repeatedly execute the same script as to execute a script that stays running. It makes sense if the script is to be a singleton, that the script itself manage ensuring only one version of itself is executing.