All Apps and Add-ons

python eventgen.py process not killed when stopping splunk ?

ahartge
Path Finder

I have downloaded the eventgen master from github, extracted to SA-Eventgen folder & also downloaded the McAfee add-on (Splunk-TA-Mcafee) which contains sample data & an eventgen.conf on my heavy forwarder.

This all works nicely, I am getting data into my main index, generated on the HF & sprayed across two indexers, but it appears that when I stop splunk on the HF - the python eventgen process is not killed / stopped.

You can see below that the child eventgen.py (PID 13292) is being called by /bin.sh (PID 13291)

When I stop Splunk - the child eventgen.py process (PID 13292) is still running.

Stopping / Starting splunk causes multiple of these processes to run, duplicating the generated data.

Has anyone seen this ? / know what could be causing it ?

root@DEV-vFWD01:~# ps -ef | grep python
root     13259 13207  0 05:00 ?        00:00:08 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
root     13291 13207  0 05:00 ?        00:00:00 /bin/sh -c python /opt/splunk/etc/apps/SA-Eventgen/bin/eventgen.py
root     13292 13291  2 05:00 ?        00:01:01 python /opt/splunk/etc/apps/SA-Eventgen/bin/eventgen.py
root     13408  8844  0 05:45 pts/0    00:00:00 grep --color=auto python
root@DEV-vFWD01:~# /opt/splunk/bin/splunk stop
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
..
Stopping splunk helpers...

Done.
root@DEV-vFWD01:~# ps -ef | grep python
root     13292     1  2 05:00 ?        00:01:02 python /opt/splunk/etc/apps/SA-Eventgen/bin/eventgen.py
root     13439  8844  0 05:46 pts/0    00:00:00 grep --color=auto python
root@DEV-vFWD01:~#
Tags (2)
1 Solution

csharp_splunk
Splunk Employee
Splunk Employee

No, your problem is the use of dash as the default shell in Ubuntu. Run the following:

debconf-set-selections <<< "dash dash/sh string false"
dpkg-reconfigure -f noninteractive dash

This will set your default shell to bash and the problem will likely go away.

View solution in original post

darthmdh
Explorer

I've filed a bug against dash in Ubuntu for this misbehaviour. Searching the internet shows its cropped up in various guises over the years depending on which application spawned dash as /bin/sh (e.g. PHP), and in every case the accepted solution was to stop using dash. As someone who isn't fond of bash (zsh FTW! 🙂 I don't think that solution is long-term acceptable.

https://bugs.launchpad.net/ubuntu/+source/dash/+bug/1473000

0 Karma

csharp_splunk
Splunk Employee
Splunk Employee

No, your problem is the use of dash as the default shell in Ubuntu. Run the following:

debconf-set-selections <<< "dash dash/sh string false"
dpkg-reconfigure -f noninteractive dash

This will set your default shell to bash and the problem will likely go away.

ahartge
Path Finder

I am marking this as correct as it resolved the problem - the question now remains as to "why" this is happening.

You can see below that indeed dash/sh was set to true, then changing it to false, I no longer get the issue.

root@splunkbox:~# debconf-show dash
* dash/sh: true
root@splunkbox:~# debconf-set-selections <<< "dash dash/sh string false"
root@splunkbox:~# debconf-show dash
* dash/sh: false

root@splunkbox:~# dpkg-reconfigure -f noninteractive dash
Removing 'diversion of /bin/sh to /bin/sh.distrib by dash'
Adding 'diversion of /bin/sh to /bin/sh.distrib by bash'
Removing 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash'
Adding 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by bash'
root@splunkbox:~# debconf-show dash
* dash/sh: false
root@splunkbox:~# 

Python is no longer called with /bin/sh -c

root@splunkbox:~# ps -ef | grep python
root      5571  5514  0 20:25 ?        00:00:00 python /opt/splunk/etc/apps/file_meta_data/bin/file_meta_data.py
root      5572  5514  1 20:25 ?        00:00:01 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
root      5640  5514  0 20:25 ?        00:00:00 python /opt/splunk/etc/apps/syndication/bin/syndication.py
root      5645  5514  0 20:25 ?        00:00:00 python /opt/splunk/etc/apps/website_input/bin/web_input.py
root      6479  1918  0 20:28 pts/0    00:00:00 grep --color=auto python
root@splunkbox:~# /opt/splunk/bin/splunk stop
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
..
Stopping splunk helpers...

Done.
root@splunkbox:~# ps -ef | grep python
root      6843  1918  0 20:29 pts/0    00:00:00 grep --color=auto python
root@splunkbox:~# 
0 Karma

ahartge
Path Finder

Support ticket has been closed & documentation updated: http://docs.splunk.com/Documentation/Splunk/6.2.4/Installation/InstallonLinux#Default_shell

Bottom line - dont use dash 😄

0 Karma

koshyk
Super Champion

i have noticed the same problem in Ubuntu 14.04. Up voting for asking it 🙂

0 Karma

ahartge
Path Finder

I have also logged a Splunk support ticket 252386, will post back once they work it out.

0 Karma

ahartge
Path Finder

Just to provide some more feedback, I built a new 14.04 machine, and I see the same problem with the syndication RSS app too https://splunkbase.splunk.com/app/2646/

I dont think it is App specific, but as you say, specific to Ubuntu 14.04 - the investigation continues.......

0 Karma

csharp_splunk
Splunk Employee
Splunk Employee

What operating system?

0 Karma

ahartge
Path Finder

Ubuntu 14.04

Although I havent tried to reproduce it on another machine, I think the eventgen was installed through the GUI then the folder was renamed from eventgen-master to SA-Eventgen afterwards - maybe this could contribute to the issue.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

I've seen this before as well. Modular Inputs in Splunk 5.X used to do it. It has something to do with the way they are spawned as inputs, and they don't check the parent process after spawning. Windows does not have this problem (oddly enough).

The only "fix" that I know of would be to edit the python file "eventgen.py" to add conditions that check the parent pid that spawned the process. If it is 1 (meaning it went to root of the system, the Splunk daemon disappeared), then exit the script.

I'll ping the author on it, maybe he can shed some additional light.

0 Karma

csharp_splunk
Splunk Employee
Splunk Employee

We already do that :).

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...