Splunkd is not running

[ZaugustZ]

Hi!

Can somebody help me with this problem?my splunkd is not running. I tried to stop and start the splunk, but splunkd runs in few seconds and stop again. I also checked /var/log/splunk/web_service.log and it shows:

2013-10-26 02:33:27,815 ERROR [526ab977d0c9603d0] startup:84 - Unable to read in product version information; Splunkd daemon is not responding: ('[Errno 111] C
onnection refused',)
2013-10-26 02:33:27,816 ERROR [526ab977d0c9603d0] decorators:383 - Splunkd daemon is not responding: ('[Errno 111] Connection refused',)
2013-10-26 02:33:43,683 ERROR [526ab987ae2aaaad1755d0] startup:84 - Unable to read in product version information; Splunkd daemon is not responding: ('[Errno 1
11] Connection refused',)
2013-10-26 02:33:43,683 ERROR [526ab987ae2aaaad1755d0] decorators:383 - Splunkd daemon is not responding: ('[Errno 111] Connection refused',)
2013-10-26 02:33:44,970 ERROR [526ab988f82aaab9436310] startup:84 - Unable to read in product version information; Splunkd daemon is not responding: ('[Errno 1
11] Connection refused',)

On my web it says:
"The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running."

Crash Log content:
[build 149561] 2013-10-29 14:53:34
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 7735 running under UID 0.
Crashing thread: MainTailingThread
Registers:
RIP: [0x0000003BED830265] gsignal + 53 (/lib64/libc.so.6)
RDI: [0x0000000000001E37]
RSI: [0x0000000000001EB6]
RBP: [0x0000000045792940]
RSP: [0x00000000457914F8]
RAX: [0x0000000000000000]
RBX: [0x00000000457915A0]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000080]
R9: [0x0101010101010101]
R10: [0x0000000000000008]
R11: [0x0000000000000202]
R12: [0x00007FFF76D5DA37]
R13: [0x0000000001307930]
R14: [0x00000000000000E5]
R15: [0x0000000001307210]
EFL: [0x0000000000000202]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace:
[0x0000003BED830265] gsignal + 53 (/lib64/libc.so.6)
[0x0000003BED831D10] abort + 272 (/lib64/libc.so.6)
[0x0000003BED8296E6] __assert_fail + 246 (/lib64/libc.so.6)
[0x00000000006FCD42] _ZN16FileInputTracker10computeCRCEPm14FileDescriptorRK3Strll + 1906 (splunkd)
[0x00000000006FCE71] _ZN16FileInputTracker11fileHalfMd5EPm14FileDescriptorRK3Strll + 17 (splunkd)
[0x000000000071B949] _ZN3WTF13loadFishStateEb + 905 (splunkd)
[0x000000000070A6C5] _ZN10TailReader8readFileER15WatchedTailFileP11TailWatcher + 149 (splunkd)
[0x000000000070A8E4] _ZN11TailWatcher8readFileER15WatchedTailFile + 260 (splunkd)
[0x000000000070C9FB] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 363 (splunkd)
[0x0000000000D3F4E1] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 113 (splunkd)
[0x0000000000D40DCF] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 479 (splunkd)
[0x0000000000DA5553] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 227 (splunkd)
[0x0000000000D3A318] _ZN9EventLoop3runEv + 216 (splunkd)
[0x000000000071328F] _ZN11TailWatcher3runEv + 143 (splunkd)
[0x00000000007133EB] _ZN13TailingThread4mainEv + 267 (splunkd)
[0x0000000000DA2F32] _ZN6Thread8callMainEPv + 66 (splunkd)
[0x0000003BEE00673D] ? (/lib64/libpthread.so.0)
[0x0000003BED8D3D1D] clone + 109 (/lib64/libc.so.6)
Linux / localhost.localdomain / 2.6.18-194.el5 / #1 SMP Fri Apr 2 14:58:14 EDT 2010 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2013-10-29 14:50:26.924 +0800 splunkd started (build 149561)
splunkd: /opt/splunk/p4/splunk/branches/5.0.2/src/pipeline/input/FileInputTracker.cpp:229: static bool FileInputTracker::computeCRC(uint64_t*, FileDescriptor, const Str&, file_offset_t, file_offset_t): Assertion bytesToHash < 1048576' failed.
2013-10-29 14:52:56.782 +0800 splunkd started (build 149561)
splunkd: /opt/splunk/p4/splunk/branches/5.0.2/src/pipeline/input/FileInputTracker.cpp:229: static bool FileInputTracker::computeCRC(uint64_t*, FileDescriptor, const Str&, file_offset_t, file_offset_t): AssertionbytesToHash < 1048576' failed.

/etc/redhat-release: CentOS release 5.5 (Final)
glibc version: 2.5
glibc release: stable
Threads running: 40
argv: [splunkd -p 8089 restart]
terminating...

Please enlighten me!
Thanks in Advance!

Regards,
ZaugustZ

[nasimm]

hi , in installing universal forwarder in search head ask me mngmnt port: and i set 8090 , now with netstat i get 8000 , 8089 , 8090 third of them are tcp , but i cant see ui of localhost:8000 , what should i do ? please help me please , please :(((
i see apache 2 in lovalost , but with port 8000 no,
what is my problem?can you help me?

[Damien_Dallimor]

Known issue : http://answers.splunk.com/answers/76178/splunkd-crash-during-startup-assertion-bytestohash-1048576-f...

[Damien_Dallimor]

If you really need to stay on version 5 , I would upgrade my 5.0.2 release to the latest version 5 release (splunk-5.0.5-179365) where the issue is patched.

[ZaugustZ]

i did some workaround in this bug but still didnt work 😞

[polymorphic]

The splunkd.log could also be interesting, but i have seen something similar on a Linux machine, where permissions were set wrong.

So i created a 'splunk' user and 'splunk' group and then did:
#chown -Rf splunk:splunk /opt/splunk
#chmod -Rf 755 /opt/splunk

But the obvious could also be the issue. If you have a firewall policy you should allow connections port 8000 (this is the default port for the splunk web) and port 8089 (default for splunkd)