topic Re: Splunk stops recording events on hosts in Getting Data In

Splunk stops recording events on hosts

gmachacek — Fri, 16 Nov 2012 15:31:33 GMT

Greetings,
Ever since upgrading to Splunk5 I've had an issue where after a random amount of time Splunk will quit logging events from some of my hosts. I have 15+ servers sending normal syslogs and running a custom app that sends it's events on 515. When this issue happens on these couple hosts, Splunk continues to receive syslog messages, but quits seeing any events from our app.

I've used tcpdump on hosts and the Splunk server; events are still being received by the server.

The port is still up and listening, because I'm still getting the app data from the other hosts.

I've looked in splunkd.log, don't see anything happening around the time I quit getting events.

It's like Splunk has decided to just ignore these particular hosts, but only on the 515 port.

Restarting splunk fixes it and everything works, for about 10-12 hours..

I did have some SEDCMD commands, but I have since commented those out and I am still see the issue.

I'm not sure where else to look....

Re: Splunk stops recording events on hosts

sdaniels — Fri, 16 Nov 2012 15:45:35 GMT

Upgrade to 5.0.1 should fix this issue.

http://docs.splunk.com/Documentation/Splunk/5.0.1/ReleaseNotes/5.0.1

Check this:

http://splunk-base.splunk.com/answers/64055/memory-usage-increase-in-v5

Re: Splunk stops recording events on hosts

gmachacek — Fri, 16 Nov 2012 18:58:57 GMT

I have update Splunk. I'll see tonight if it has (hopefully) fixed the issue. 😉

Thanks!

Re: Splunk stops recording events on hosts

gmachacek — Mon, 19 Nov 2012 16:36:25 GMT

Greetings,
Well it increased the time before it quit recording entries for those hosts. It made it 48 hours versus 8-10 hours.

Running splunk-5.0.1-143156

Re: Splunk stops recording events on hosts

sdaniels — Mon, 19 Nov 2012 20:27:48 GMT

I would recommend contacting support, perhaps the fix does not quite cover all possible cases. Also it is a best practise to send your data into an rsyslog/syslog-ng first and then have splunk monitor those files.