topic Re: Crashing thread: OneShotFileWriter in Monitoring Splunk

Crashing thread: OneShotFileWriter

rroberts — Mon, 28 Sep 2020 15:58:16 GMT

Batch upload a diag : 5178152 2011-11-17 17:18 diag-ip-10-168-65-59-2011-11-17.tar.gz (not too big I don't think)
Splunkd crashed. Any Idea what may have caused the crash?

Crash log snippet indicates the OneShotFileWriter thread is the culprit but have no idea why.

[build 189883] 2014-02-24 15:34:15
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 1972 running under UID 521.
Crashing thread: OneShotFileWriter
OS: Linux
Arch: x86-64

Backtrace:
[0x00002AAAABC01EC5] gsignal + 53 (/lib64/libc.so.6)
[0x00002AAAABC03970] abort + 272 (/lib64/libc.so.6)
[0x00002AAAABBFB11F] __assert_fail + 239 (/lib64/libc.so.6)
[0x0000000000B4505F] _ZN20PipelineInputChannel16addNumberedFieldERK10StrSegment + 159 (splunkd)
[0x00002AAAAF020F01] ?
Linux / ip-10-188-188-218 / 2.6.21.7-2.ec2.v1.2.fc8xen / #1 SMP Fri Nov 20 17:48:28 EST 2009 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):

Splunk std err shows…
2014-02-24 10:45:22.405 -0500 splunkd started (build 189883)
Cannot open manifest file inside "/home/stu4/opt/splunk/var/lib/splunk/defaultdb/db/db_1311012880_1283850332_0/rawdata": No such file or directory
Cannot open manifest file inside "/home/stu4/opt/splunk/var/lib/splunk/defaultdb/db/db_1373425154_1373415809_2/rawdata": No such file or directory
splunkd: /opt/splunk/p4/splunk/branches/6.0.1/src/framework/PipelineInputChannel.cpp:706: void PipelineInputChannel::addNumberedField(const StrSegment&): Assertion `_refcnt == 1' failed.
2014-02-24 15:36:18.312 -0500 splunkd started (build 189883)
Cannot open manifest file inside "/home/stu4/opt/splunk/var/lib/splunk/23456/db/db_1393265100_1393265100_2/rawdata": No such file or directory

[stu4@ip-10-188-188-218 bin]$ ./splunk version
Splunk 6.0.1 (build 189883)

[root@ip-10-188-188-218 splunker]# uname -a
Linux ip-10-188-188-218 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

Re: Crashing thread: OneShotFileWriter

lukejadamec — Tue, 25 Feb 2014 16:41:48 GMT

Do the manifest files exist? Does user UID 521 have access to the manifest files?

Re: Crashing thread: OneShotFileWriter

rroberts — Wed, 26 Feb 2014 23:12:50 GMT

Unfortunately this training instance is gone. I'll spin up a new one and see if I can recreate.

Re: Crashing thread: OneShotFileWriter

rroberts — Wed, 03 Sep 2014 18:28:31 GMT

Happened again today. This time UID 526. Manifest file shows:

-r--r--r-- 1 instructor student 840213 Sep 2 16:57 splunk-6.1.1-207789-linux-2.6-x86_64-manifest

Splunk 6.1.1 by the way.

Re: Crashing thread: OneShotFileWriter

rroberts — Wed, 17 Sep 2014 20:13:15 GMT

And happened again today:
Signal sent by PID 1849 running under UID 514.
Crashing thread: OneShotFileWriter
Im running Splunk as UID 514, I created the diag as splunk admin (UID 514) . Then attempted to eat the diag via SplunkWeb.
The manifest file shows:
f 444 splunk splunk splunk/README-splunk.txt

d 755 splunk splunk splunk/bin

f 555 splunk splunk splunk/bin/bloom

f 555 splunk splunk splunk/bin/btool

f 555 splunk splunk splunk/bin/btprobe

Re: Crashing thread: OneShotFileWriter

sowings — Wed, 17 Sep 2014 21:19:03 GMT

The manifest file in question is the one from the data bucket, not the top level manifest. Is that index (23456) healthy?

Re: Crashing thread: OneShotFileWriter

rroberts — Thu, 11 Dec 2014 21:17:03 GMT

It was a newly created index with nothing in it.

Re: Crashing thread: OneShotFileWriter

rroberts — Thu, 11 Dec 2014 21:17:36 GMT

Problem seemed to go away after upgrading to 6.2! Woohoo!