All Apps and Add-ons

Why is lsof_sos.sh not returning any data?

Jason
Motivator

We have just deployed TA-sos to all search heads and indexers. Both inputs (ps_sos.sh and lsof_sos.sh) are enabled, but no lsof_sos source data is being received. Running the script manually, it produces no output. Any idea what is going wrong?

lsof is not on path, but even editing the script to call lsof at its actual location /usr/sbin/lsof still produces no output.

/usr/sbin/lsof -n -P -s -p [splunkd pid from ./splunk status],[splunkweb pid from ./splunk status] produces plenty of output, even run as the splunk user.

asimagu
Builder

it may be a problem of $SPLUNK_HOME not having been set, therefore when the scripts tries to find that variable, it is not able to find it

0 Karma

heybigben
Explorer

UPDATE: This is fixed in S.o.S 3.1 and will be fixed in the next release of the S.o.S addon for Linux and Unix (2.0.5, in all likelihood).

We had the same issue where lsof data was not showing up in splunk. This problem was happening in both the unix app and the SOS app despite the inputs being enabled. On our rhel5.8 system lsof is at /usr/sbin/lsof but both splunk apps - in common.sh for linux systems only set the path to /sbin which resulted in the lsof command not being found and thus no data was being returned. As a temp workaround I set common.sh to have the following for the path and lsof data started showing up and the graphs generated. PATH=$PATH:/sbin/:/usr/sbin/ We will be submitting a ticket to splunk support soon to get this fixed or see if there is a better solution.

hexx
Splunk Employee
Splunk Employee

We expect to put out a new version including this fix in the next few weeks.

0 Karma

yoho
Contributor

Hi ! This problem still exists. It's easy to fix if you have a deployment server but I think apps should work correctly by default without any tuning, so waiting desperately for a 2.0.5 release of TA-sos...

0 Karma

hexx
Splunk Employee
Splunk Employee

Thanks for sharing your investigation, @heybigben! We'll assess your findings within the context of the bug currently filed against this issue (SUP-649).
If anyone else experiencing this problem is able to resolve it using @heybigben's work-around, I'd love to hear about it.

0 Karma

hexx
Splunk Employee
Splunk Employee

Thank you for the information provided. I have opened a bug against the S.o.S app (internal reference: SUP-649) to have this investigated and fixed. Hopefully, we can easily reproduce this in-house.

0 Karma

jrodman
Splunk Employee
Splunk Employee

answers is pretty awkward for debugging; but i would try (whichever the script asks for, bash or sh)

$ bash -x -v lsof_sos.sh

If you're in an fairly sensitive environment you may want to do some quick greps on the otput -- maybe hostnames may show up.

0 Karma

hexx
Splunk Employee
Splunk Employee

And just to be clear: Does running /usr/sbin/lsof -n -P -s -p <splunkd PID> manually as the splunk user yield output as expected?

0 Karma

Jason
Motivator

Thanks Laks. Yes Hexx Splunk is running as the splunk user. It's running on Oracle enterprise linux, apparently a rebadged RHEL 5.6.

0 Karma

lakshman237
Path Finder

output the other command.

[splunk@dyl10639app21 bin]$ /opt/splunk/bin/splunk cmd /opt/splunk/etc/apps/sos/bin/lsof_sos.sh
[splunk@dyl10639app21 bin]$ cd /opt/splunk/etc/apps/sos/bin/
[splunk@dyl10639app21 bin]$ ./lsof_sos.sh

sorry had to send the results in comment and in answer as i am unable to post more. I work with Jason on this .

0 Karma

lakshman237
Path Finder

[splunk@dyl10639app21 bin]$ /usr/sbin/lsof -v
lsof version information:
revision: 4.78
latest revision: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/
latest FAQ: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ
latest man page: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/lsof_man
constructed: Wed Jun 6 05:06:54 EDT 2007
constructed by and on: mockbuild@ca-build14
compiler: cc
compiler version: 4.1.1 20070105 (Red Hat 4.1.1-52)
compiler flags: -DLINUXV=26016 -DGLIBCV=205 -DHASIPv6 -DHASSELINUX -D_FILE_OFFSET_BITS=64 -DLSOF_VSTR="2.6.16" -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
loader flags: -L./lib -llsof -lselinux
system info: Linux ca-build14 2.6.20-1.2952.fc6 #1 SMP Wed May 16 18:18:22 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
Anyone can list all files.
/dev warnings are disabled.
Kernel ID check is disabled.

0 Karma

lakshman237
Path Finder

Hi, its Oracle Enterprise Linux
uname -a
Linux dyl10639app21 2.6.18-238.el5 #1 SMP Tue Jan 4 15:41:11 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

0 Karma

hexx
Splunk Employee
Splunk Employee

Hi, Jason. Could you tell us what OS and distribution this is running on? Also, could you provide the output of the following commands?

# /usr/sbin/lsof -v
# $SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/etc/apps/sos/bin/lsof_sos.sh

Also, is splunkd running as root or as a dedicated user?

0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...