Splunk stops recording events on hosts

[gmachacek]

Greetings,
Ever since upgrading to Splunk5 I've had an issue where after a random amount of time Splunk will quit logging events from some of my hosts. I have 15+ servers sending normal syslogs and running a custom app that sends it's events on 515. When this issue happens on these couple hosts, Splunk continues to receive syslog messages, but quits seeing any events from our app.

I've used tcpdump on hosts and the Splunk server; events are still being received by the server.

The port is still up and listening, because I'm still getting the app data from the other hosts.

I've looked in splunkd.log, don't see anything happening around the time I quit getting events.

It's like Splunk has decided to just ignore these particular hosts, but only on the 515 port.

Restarting splunk fixes it and everything works, for about 10-12 hours..

I did have some SEDCMD commands, but I have since commented those out and I am still see the issue.

I'm not sure where else to look....

[sdaniels]

Upgrade to 5.0.1 should fix this issue.

http://docs.splunk.com/Documentation/Splunk/5.0.1/ReleaseNotes/5.0.1

Check this:

http://splunk-base.splunk.com/answers/64055/memory-usage-increase-in-v5

[sdaniels]

I would recommend contacting support, perhaps the fix does not quite cover all possible cases. Also it is a best practise to send your data into an rsyslog/syslog-ng first and then have splunk monitor those files.

[gmachacek]

Greetings,
Well it increased the time before it quit recording entries for those hosts. It made it 48 hours versus 8-10 hours.

Running splunk-5.0.1-143156

[gmachacek]

I have update Splunk. I'll see tonight if it has (hopefully) fixed the issue. 😉

Thanks!