Getting Data In

Splunk stops recording events on hosts

gmachacek
Engager

Greetings,
Ever since upgrading to Splunk5 I've had an issue where after a random amount of time Splunk will quit logging events from some of my hosts. I have 15+ servers sending normal syslogs and running a custom app that sends it's events on 515. When this issue happens on these couple hosts, Splunk continues to receive syslog messages, but quits seeing any events from our app.

I've used tcpdump on hosts and the Splunk server; events are still being received by the server.

The port is still up and listening, because I'm still getting the app data from the other hosts.

I've looked in splunkd.log, don't see anything happening around the time I quit getting events.

It's like Splunk has decided to just ignore these particular hosts, but only on the 515 port.

Restarting splunk fixes it and everything works, for about 10-12 hours..

I did have some SEDCMD commands, but I have since commented those out and I am still see the issue.

I'm not sure where else to look....

Tags (2)
0 Karma

sdaniels
Splunk Employee
Splunk Employee

sdaniels
Splunk Employee
Splunk Employee

I would recommend contacting support, perhaps the fix does not quite cover all possible cases. Also it is a best practise to send your data into an rsyslog/syslog-ng first and then have splunk monitor those files.

0 Karma

gmachacek
Engager

Greetings,
Well it increased the time before it quit recording entries for those hosts. It made it 48 hours versus 8-10 hours.

Running splunk-5.0.1-143156

0 Karma

gmachacek
Engager

I have update Splunk. I'll see tonight if it has (hopefully) fixed the issue. 😉

Thanks!

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...