Splunk Enterprise

splunkforwarder service status is not running, but the splunkforwarder service is generating logs

BKR
Loves-to-Learn Lots

Hi 

looking for help , splunkforwarder service status is not running, but the splunkforwarder service is generating logs and we are able see the internal logs in cli  as well as in ui, need to know which cause it's showing splunk service is not running. 

splunk@:/opt/splunkforwarder/bin$ ./splunk status
splunkd is not running.
splunk@:/opt/splunkforwarder/bin$ ./splunk version
Splunk Universal Forwarder 6.5.2
splunk@:/opt/splunkforwarder/bin$ ps -ef|grep splunkd
splunk 15913 1 3 06:01 ? 00:11:54 splunkd -p 8089 start
splunk 15914 15913 0 06:01 ? 00:00:00 [splunkd pid=15913] splunkd -p 8089 start [process-runner]
splunk 18627 18439 0 11:13 pts/1 00:00:00 grep splunkd
splunk@:/opt/splunkforwarder/bin$ date
Thu Feb 18 11:14:20 UTC 2021
splunk@:/opt/splunkforwarder/bin$ cd /opt/splunkforwarder/var/log/splunk/
splunk@:/opt/splunkforwarder/var/log/splunk$ tail -5 splunkd.log
02-18-2021 11:14:31.788 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_XXXXXXX_8089_XXXXXXXXX_0D56BD7C-435F-43C3-9C3D-46732D507E0C
02-18-2021 11:14:40.477 +0000 INFO WatchedFile - Resetting fd to re-extract header.
02-18-2021 11:14:43.716 +0000 INFO TcpOutputProc - Connected to idx=XXXXXX:9998 using ACK.
02-18-2021 11:14:43.892 +0000 INFO TcpOutputProc - Connected to idx=XXXXXX:9999 using ACK.
02-18-2021 11:14:46.292 +0000 INFO TcpOutputProc - Connected to idx=XXXXXX:9999 using ACK.
splunk@:/opt/splunkforwarder/var/log/splunk$




Labels (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

You could try lsof  -p <splunk pid>  to see from which binary it’s running and which files it’s using.

0 Karma

mishra321
Loves-to-Learn

Hi,

I am also facing similar issue and ran lsof -p <splunkd pid> and I'm getting below output:-

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
splunkd 43466 splunk cwd DIR 252,0 4096 2 /
splunkd 43466 splunk rtd DIR 252,0 4096 2 /
splunkd 43466 splunk txt REG 252,11 35421040 11564 /opt/splunkforwarder/bin/splunkd
splunkd 43466 splunk mem REG 252,0 1868984 1029 /lib/x86_64-linux-gnu/libc-2.23.so
splunkd 43466 splunk mem REG 252,0 138696 1133 /lib/x86_64-linux-gnu/libpthread-2.23.so
splunkd 43466 splunk mem REG 252,0 1088952 16547 /lib/x86_64-linux-gnu/libm-2.23.so
splunkd 43466 splunk mem REG 252,11 188904 11534 /opt/splunkforwarder/lib/libbson-1.0.so.0.0.0
splunkd 43466 splunk mem REG 252,11 310152 11495 /opt/splunkforwarder/lib/libmongoc-1.0.so.0.0.0
splunkd 43466 splunk mem REG 252,11 1025736 11502 /opt/splunkforwarder/lib/libsqlite3.so.0.8.6
splunkd 43466 splunk mem REG 252,11 662808 11493 /opt/splunkforwarder/lib/libarchive.so.13.2.2
splunkd 43466 splunk mem REG 252,0 14608 1027 /lib/x86_64-linux-gnu/libdl-2.23.so
splunkd 43466 splunk mem REG 252,11 2974768 11497 /opt/splunkforwarder/lib/libcrypto.so.1.0.0
splunkd 43466 splunk mem REG 252,11 335432 11531 /opt/splunkforwarder/lib/libxmlsec1-openssl.so.1.2.20
splunkd 43466 splunk mem REG 252,11 499224 11512 /opt/splunkforwarder/lib/libxmlsec1.so.1.2.20
splunkd 43466 splunk mem REG 252,11 1855304 11530 /opt/splunkforwarder/lib/libxml2.so.2.9.4
splunkd 43466 splunk mem REG 252,0 31712 16568 /lib/x86_64-linux-gnu/librt-2.23.so
splunkd 43466 splunk mem REG 252,0 162632 16559 /lib/x86_64-linux-gnu/ld-2.23.so
splunkd 43466 splunk mem REG 252,11 108056 11483 /opt/splunkforwarder/lib/libz.so.1.2.8
splunkd 43466 splunk mem REG 252,11 73648 11501 /opt/splunkforwarder/lib/libbz2.so.1.0.3
splunkd 43466 splunk mem REG 252,11 481112 11504 /opt/splunkforwarder/lib/libssl.so.1.0.0
splunkd 43466 splunk mem REG 252,11 343080 11496 /opt/splunkforwarder/lib/libxslt.so.1.1.29
splunkd 43466 splunk mem REG 252,11 606680 11532 /opt/splunkforwarder/lib/libpcre2-8.so
splunkd 43466 splunk mem REG 252,11 319256 11487 /opt/splunkforwarder/lib/libjemalloc.so.2
splunkd 43466 splunk 0r CHR 1,3 0t0 6 /dev/null
splunkd 43466 splunk 1u REG 252,11 28798 11654 /opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
splunkd 43466 splunk 2u REG 252,11 24907 11656 /opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
splunkd 43466 splunk 3r REG 0,4 0 19849 /proc/sys/vm/overcommit_memory

what can I check over here ?

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...