Splunk Add-on for F5 BIG-IP: Is it possible to NOT start Splunk_TA_f5_bigip_main.py?

[jgoddard]

I do not seem to be able to prevent this python script from starting, and as we do not use any of the API connections, it is unnecessary. Additionally, the script launches many copies of itself which do not get shut down when Splunk restarts. I routinely end up with many of these processes left behind after splunk stops:

splunk     533  0.0  0.1  44088 23948 ?        S    Aug25   5:11 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk     574  0.0  0.1  44076 23952 ?        S    Aug23   5:35 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    2215  0.0  0.1  44068 23948 ?        S    Aug26  13:20 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    2624  0.0  0.1  43828 23752 ?        S    Sep17  14:21 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    3241  0.0  0.1  43816 23744 ?        S    Sep04  10:54 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    5024  0.0  0.1  44076 23960 ?        S    Aug26   6:00 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    6351  0.0  0.1  44076 23960 ?        S    Sep16   4:07 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    7133  0.1  0.1  43820 23484 ?        S    18:46   0:00 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    8910  0.0  0.0   4440   652 ?        Ss   18:55   0:00 /bin/sh -c python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py
splunk    8911  0.3  0.1  43812 23484 ?        S    18:55   0:00 python /opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py

Thanks.

[jcoates_splunk]

Are you on Debian or Ubuntu? There's a known issue with /bin/dash and SIGTERM that can cause problems. What do you see in this search?

index=_internal Splunk_TA_f5_bigip_main.py

[jcoates_splunk]

we're still working on this one... please file tickets and ask for reference to SPL-104398 if you're seeing this problem.

[jcoates_splunk]

the best option at this point is to use debconf to change your default shell to bash.

debconf-set-selections <<< "dash dash/sh string false"
dpkg-reconfigure -f noninteractive dash

the second best option is to edit local/inputs.conf and specify

start_by_shell = false

This is suboptimal because it has to be done for all modular inputs and will produce startup warnings, but it should stop the unintentional process spawning.

[jgoddard]

Realized i did not specifically answer your question about the OS. We are using ubuntu for these devices.

[jcoates_splunk]

that means you've probably got lots of Splunk_TA_f5_bigip_main.py processes in memory preventing things from working right. A support case might be useful so we can keep you updated, we're still working on the least intrusive way to do the same thing on RHEL and Ubuntu without asking people to switch shells.

[jgoddard]

yes, yes i do.

I will open a case about this for updates, but I am willing to switch the shell that my splunk user gets, if you have a suggestion that is less broken than dash.

[jcoates_splunk]

we develop and test on RHEL and CentOS, which is /bin/bash. The main loop here is launching as part of a validation process which is supposed to immediate exit if you don't use the API calls; that's something we'll want to change anyway, but in the meantime avoiding dash might help.

[jgoddard]

I seem to get one error routinely:
2015-09-29 20:44:28,971 ERROR pid=8318 tid=MainThread file=Splunk_TA_f5_bigip_main.py:stream_events:117 | Error in getting task definitions by restful API(maybe splunk daemon is down?) - Traceback:
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip_main.py", line 104, in _stream_events
server_manager.reload()
File "/opt/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/Modules/F5Servers.py", line 297, in reload
splunk.rest.simpleRequest(F5ServerModel.build_id(None, None, None) + "/_reload", sessionKey=self._sessionKey)
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init_.py", line 508, in simpleRequest
raise splunk.AuthenticationFailed
AuthenticationFailed: [HTTP 401] Client is not authenticated