All Apps and Add-ons

How can I get Splunk_TA_nix to stop running lsof.sh?

esalesapns2
Path Finder

I can't figure out why lsof.sh is running every minute.
Here's the
"btool inputs list --debug" output for lsof:

/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf disabled = 1
/opt/splunkforwarder/etc/system/local/inputs.conf host = c20sbap01l01
/opt/splunkforwarder/etc/apps/DS2-ns2-Splunk_TA_nix-cre/local/inputs.conf index = cre_linux
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf interval = 600
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf source = lsof
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf sourcetype = lsof

Here's my splund.log output:

10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh
10-10-2019 16:07:12.898 +0000 INFO ExecProcessor - interval: 60000 ms

I've tried restarting splunk to no effect...
Notice that the interval is set to 600 (600 seconds) in the btool output, but 60000 (60 seconds) in the splunkd.log output.
I'll try interval = -1 next, and then a single app after that.

Labels (1)
0 Karma
1 Solution

esalesapns2
Path Finder

I had to remove/re-install the splunk_TA_nix app to get it to stop behaving this way.

View solution in original post

0 Karma

esalesapns2
Path Finder

I had to remove/re-install the splunk_TA_nix app to get it to stop behaving this way.

0 Karma

corybits
New Member

Thank you for the insights shared, my question is; when you set the indexes for all the Splunk_TA_nix inputs in Splunk_TA_nix/local/inputs.conf, did you use the same index for all the inputs, and was the index/indexes defined in your indexes.conf? Bare with me, new to Splunk.

0 Karma

woodcock
Esteemed Legend

The btool is your friend here. You could have an inputs.conf in any app that is causing things so try this:

$SPLUNK_HOME/bin/splunk btool list inputs --debug | grep lsof
0 Karma

esalesapns2
Path Finder

Ok, I stopped Splunk, removed the Splunk_TA_nix app, started splunk put the app back and started splunk, and I'm finally no longer getting lsof events. However, I now need to do the same on all my deployment clients... Good thing I was planning on working late.

0 Karma

esalesapns2
Path Finder

I should say I restarted Splunk after I put the app back.

0 Karma

jacobpevans
Motivator

Dumb question, but did you run btool on the same machine that the splunkd log is from?

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

esalesapns2
Path Finder

Yes, I did. I have since stopped Splunk, removed deploymentclient.conf, and the DS2-Splunk_TA_nix-cre directory, and set the indexes for all the Splunk_TA_nix inputs in Splunk_TA_nix/local/inputs.conf, and restarted splunk. I'm still getting a steady stream of lsof events every minute.

0 Karma

esalesapns2
Path Finder

This is on Splunk Universal Forwarder 7.0.1.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Do you want to stop lsof.sh from running at all or just make it run every 10 minutes?

---
If this reply helps you, Karma would be appreciated.
0 Karma

esalesapns2
Path Finder

stop it altogether. some of my servers have 5M files open at a time...

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...