Monitoring Splunk

Splunkd is not running

ZaugustZ
Explorer

Hi!

Can somebody help me with this problem?my splunkd is not running. I tried to stop and start the splunk, but splunkd runs in few seconds and stop again. I also checked /var/log/splunk/web_service.log and it shows:

2013-10-26 02:33:27,815 ERROR [526ab977d0c9603d0] startup:84 - Unable to read in product version information; Splunkd daemon is not responding: ('[Errno 111] C
onnection refused',)
2013-10-26 02:33:27,816 ERROR [526ab977d0c9603d0] decorators:383 - Splunkd daemon is not responding: ('[Errno 111] Connection refused',)
2013-10-26 02:33:43,683 ERROR [526ab987ae2aaaad1755d0] startup:84 - Unable to read in product version information; Splunkd daemon is not responding: ('[Errno 1
11] Connection refused',)
2013-10-26 02:33:43,683 ERROR [526ab987ae2aaaad1755d0] decorators:383 - Splunkd daemon is not responding: ('[Errno 111] Connection refused',)
2013-10-26 02:33:44,970 ERROR [526ab988f82aaab9436310] startup:84 - Unable to read in product version information; Splunkd daemon is not responding: ('[Errno 1
11] Connection refused',)

On my web it says:
"The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running."

Crash Log content:
[build 149561] 2013-10-29 14:53:34
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 7735 running under UID 0.
Crashing thread: MainTailingThread
Registers:
RIP: [0x0000003BED830265] gsignal + 53 (/lib64/libc.so.6)
RDI: [0x0000000000001E37]
RSI: [0x0000000000001EB6]
RBP: [0x0000000045792940]
RSP: [0x00000000457914F8]
RAX: [0x0000000000000000]
RBX: [0x00000000457915A0]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000080]
R9: [0x0101010101010101]
R10: [0x0000000000000008]
R11: [0x0000000000000202]
R12: [0x00007FFF76D5DA37]
R13: [0x0000000001307930]
R14: [0x00000000000000E5]
R15: [0x0000000001307210]
EFL: [0x0000000000000202]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace:
[0x0000003BED830265] gsignal + 53 (/lib64/libc.so.6)
[0x0000003BED831D10] abort + 272 (/lib64/libc.so.6)
[0x0000003BED8296E6] __assert_fail + 246 (/lib64/libc.so.6)
[0x00000000006FCD42] _ZN16FileInputTracker10computeCRCEPm14FileDescriptorRK3Strll + 1906 (splunkd)
[0x00000000006FCE71] _ZN16FileInputTracker11fileHalfMd5EPm14FileDescriptorRK3Strll + 17 (splunkd)
[0x000000000071B949] _ZN3WTF13loadFishStateEb + 905 (splunkd)
[0x000000000070A6C5] _ZN10TailReader8readFileER15WatchedTailFileP11TailWatcher + 149 (splunkd)
[0x000000000070A8E4] _ZN11TailWatcher8readFileER15WatchedTailFile + 260 (splunkd)
[0x000000000070C9FB] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 363 (splunkd)
[0x0000000000D3F4E1] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 113 (splunkd)
[0x0000000000D40DCF] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 479 (splunkd)
[0x0000000000DA5553] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 227 (splunkd)
[0x0000000000D3A318] _ZN9EventLoop3runEv + 216 (splunkd)
[0x000000000071328F] _ZN11TailWatcher3runEv + 143 (splunkd)
[0x00000000007133EB] _ZN13TailingThread4mainEv + 267 (splunkd)
[0x0000000000DA2F32] _ZN6Thread8callMainEPv + 66 (splunkd)
[0x0000003BEE00673D] ? (/lib64/libpthread.so.0)
[0x0000003BED8D3D1D] clone + 109 (/lib64/libc.so.6)
Linux / localhost.localdomain / 2.6.18-194.el5 / #1 SMP Fri Apr 2 14:58:14 EDT 2010 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2013-10-29 14:50:26.924 +0800 splunkd started (build 149561)
splunkd: /opt/splunk/p4/splunk/branches/5.0.2/src/pipeline/input/FileInputTracker.cpp:229: static bool FileInputTracker::computeCRC(uint64_t*, FileDescriptor, const Str&, file_offset_t, file_offset_t): Assertion bytesToHash < 1048576' failed.
2013-10-29 14:52:56.782 +0800 splunkd started (build 149561)
splunkd: /opt/splunk/p4/splunk/branches/5.0.2/src/pipeline/input/FileInputTracker.cpp:229: static bool FileInputTracker::computeCRC(uint64_t*, FileDescriptor, const Str&, file_offset_t, file_offset_t): Assertion
bytesToHash < 1048576' failed.

/etc/redhat-release: CentOS release 5.5 (Final)
glibc version: 2.5
glibc release: stable
Threads running: 40
argv: [splunkd -p 8089 restart]
terminating...

Please enlighten me!
Thanks in Advance!

Regards,
ZaugustZ

Tags (2)

nasimm
New Member

hi , in installing universal forwarder in search head ask me mngmnt port: and i set 8090 , now with netstat i get 8000 , 8089 , 8090 third of them are tcp , but i cant see ui of localhost:8000 , what should i do ? please help me please , please :(((
i see apache 2 in lovalost , but with port 8000 no,
what is my problem?can you help me?

0 Karma

Damien_Dallimor
Ultra Champion
0 Karma

Damien_Dallimor
Ultra Champion

If you really need to stay on version 5 , I would upgrade my 5.0.2 release to the latest version 5 release (splunk-5.0.5-179365) where the issue is patched.

0 Karma

ZaugustZ
Explorer

i did some workaround in this bug but still didnt work 😞

0 Karma

polymorphic
Communicator

The splunkd.log could also be interesting, but i have seen something similar on a Linux machine, where permissions were set wrong.

So i created a 'splunk' user and 'splunk' group and then did:
#chown -Rf splunk:splunk /opt/splunk
#chmod -Rf 755 /opt/splunk

But the obvious could also be the issue. If you have a firewall policy you should allow connections port 8000 (this is the default port for the splunk web) and port 8089 (default for splunkd)

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...