I went through and cleaned up old alerts in saved searches and looks for funky characters, but couldn't get it to come up.
I ended up spinning up a new server, and installed splunk 6.4.5 on it
I pointed it at the existing splunk license master, and then added all the 6.5.1 indexers to it.
From the original search head I copied the /opt/splunk/etc/apps/search folder over.
I had a back up of that folder before I upgraded to 6.5.1, not sure if that would have caused issues if i had not had the old files.
It complains and says searching won't work (since indexers were on 6.5.1 and the search head is 6.4.5) but I have all my Production Alert/Reports working again. So I can at least get by until this patch.
... View more
Ever since upgrading to Splunk5 I've had an issue where after a random amount of time Splunk will quit logging events from some of my hosts. I have 15+ servers sending normal syslogs and running a custom app that sends it's events on 515. When this issue happens on these couple hosts, Splunk continues to receive syslog messages, but quits seeing any events from our app.
I've used tcpdump on hosts and the Splunk server; events are still being received by the server.
The port is still up and listening, because I'm still getting the app data from the other hosts.
I've looked in splunkd.log, don't see anything happening around the time I quit getting events.
It's like Splunk has decided to just ignore these particular hosts, but only on the 515 port.
Restarting splunk fixes it and everything works, for about 10-12 hours..
I did have some SEDCMD commands, but I have since commented those out and I am still see the issue.
I'm not sure where else to look....
... View more