https://community.splunk.com/t5/Monitoring-Splunk/Splunkd-fatal-error/m-p/60336#M679 <P>Recently I migrated the Windows Splunk server in our QA environment to Ubuntu 10.04.</P> <P>Things were working well for a week, today splunkd crashed. I suspect it is because the open file limit was set to a low number on the linux server. I have increased the open file limit and restarted splunkd.</P> <P>Looking at the logs, can anyone confirm if this theory is true? If not, any thoughts on why this happened? Thanks.</P> <P>Tailing <STRONG>Splunkd.log</STRONG>, the last error messages before the crash (after a bunch of info messages) are:</P> <P>08-18-2011 15:40:31.187 +0000 ERROR JournalSlice - Cannot create new journal slice file: Too many open files, file="/opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_42901/rawdata/0"</P> <P>08-18-2011 15:40:31.188 +0000 ERROR JournalSlice - Failed to write header for rawdata</P> <P>08-18-2011 15:40:31.188 +0000 INFO HotDBManager - no hot found for event ts=1303840686, closest match=id=42900 [et,lt,span,flush,lru]=[1303676631,1303676631,14400,9223372036854775807,1313682031] [expanded span=164055]</P> <P>08-18-2011 15:40:31.188 +0000 FATAL HotDBManager - hot dir with id already exists in createDir: /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_42901</P> <P>08-18-2011 15:40:31.357 +0000 WARN EventLoop - Main Thread: about to throw a EventLoopException: error from PolledSocket write: Broken pipe</P> <P>Tailing <STRONG>/var/log/messages</STRONG>:</P> <P>Aug 18 06:41:13 QAIFSPLUNK02 rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="790" x-info="<A href="http://www.rsyslog.com%22" target="_blank">http://www.rsyslog.com"</A>] rsyslogd was HUPed, type 'lightweight'.</P> <P>Aug 18 15:40:31 QAIFSPLUNK02 kernel: [781860.711631] __ratelimit: 3 callbacks suppressed</P> <P>Aug 18 15:40:31 QAIFSPLUNK02 kernel: [781860.711641] splunkd[12731]: segfault at 157a000 ip 0000000000f33280 sp 00007fb9ff7b90d0 error 4 in splunkd[400000+1017000]</P> Mon, 28 Sep 2020 09:48:29 GMT sdevadas 2020-09-28T09:48:29Z https://community.splunk.com/t5/Monitoring-Splunk/Splunkd-fatal-error/m-p/60336#M679 <P>Recently I migrated the Windows Splunk server in our QA environment to Ubuntu 10.04.</P> <P>Things were working well for a week, today splunkd crashed. I suspect it is because the open file limit was set to a low number on the linux server. I have increased the open file limit and restarted splunkd.</P> <P>Looking at the logs, can anyone confirm if this theory is true? If not, any thoughts on why this happened? Thanks.</P> <P>Tailing <STRONG>Splunkd.log</STRONG>, the last error messages before the crash (after a bunch of info messages) are:</P> <P>08-18-2011 15:40:31.187 +0000 ERROR JournalSlice - Cannot create new journal slice file: Too many open files, file="/opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_42901/rawdata/0"</P> <P>08-18-2011 15:40:31.188 +0000 ERROR JournalSlice - Failed to write header for rawdata</P> <P>08-18-2011 15:40:31.188 +0000 INFO HotDBManager - no hot found for event ts=1303840686, closest match=id=42900 [et,lt,span,flush,lru]=[1303676631,1303676631,14400,9223372036854775807,1313682031] [expanded span=164055]</P> <P>08-18-2011 15:40:31.188 +0000 FATAL HotDBManager - hot dir with id already exists in createDir: /opt/splunk/var/lib/splunk/defaultdb/db/hot_v1_42901</P> <P>08-18-2011 15:40:31.357 +0000 WARN EventLoop - Main Thread: about to throw a EventLoopException: error from PolledSocket write: Broken pipe</P> <P>Tailing <STRONG>/var/log/messages</STRONG>:</P> <P>Aug 18 06:41:13 QAIFSPLUNK02 rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="790" x-info="<A href="http://www.rsyslog.com%22" target="_blank">http://www.rsyslog.com"</A>] rsyslogd was HUPed, type 'lightweight'.</P> <P>Aug 18 15:40:31 QAIFSPLUNK02 kernel: [781860.711631] __ratelimit: 3 callbacks suppressed</P> <P>Aug 18 15:40:31 QAIFSPLUNK02 kernel: [781860.711641] splunkd[12731]: segfault at 157a000 ip 0000000000f33280 sp 00007fb9ff7b90d0 error 4 in splunkd[400000+1017000]</P> Mon, 28 Sep 2020 09:48:29 GMT https://community.splunk.com/t5/Monitoring-Splunk/Splunkd-fatal-error/m-p/60336#M679 sdevadas 2020-09-28T09:48:29Z https://community.splunk.com/t5/Monitoring-Splunk/Splunkd-fatal-error/m-p/60337#M680 <P>I would suggest that you restart splunk to roll this particular hotbucket to warm, it looks like something has gone haywire in your hot bucket. </P> Wed, 12 Oct 2011 17:21:27 GMT https://community.splunk.com/t5/Monitoring-Splunk/Splunkd-fatal-error/m-p/60337#M680 jbsplunk 2011-10-12T17:21:27Z