Deployment Architecture

Splunk stops logging, requires service restart

Explorer

We have splunk logging via inbound syslog stream. Recently, we have found that Splunk is no longer logging the data (doing a wildcard search in realtime, no entries appear). We restart the service (splunk stop/start) and that seems to kick it back into gear. This is starting to happen more and more frequently. Any ideas where to look to determine why this is happening and how to resolve?

We are running splunk 4.1.6 on 64 bit windows server.

0 Karma
1 Solution

Explorer

The issue was not that Splunk stopped indexing. It turned out to be that the timestamp of the current time changed to a year in the past. I will close this post and open another to try to determine why the current timestamp changed and how to workaround with props.conf.

View solution in original post

0 Karma

Explorer

The issue was not that Splunk stopped indexing. It turned out to be that the timestamp of the current time changed to a year in the past. I will close this post and open another to try to determine why the current timestamp changed and how to workaround with props.conf.

View solution in original post

0 Karma

Explorer

We have determined that whatever is causing indexing to stop/crash is likely due to the next message coming in via syslog port. This message does not actually get written to the index and it does not seem to trigger anything in the splunkd.log. Is there any other locations I can look to see if the contents of this message are getting recorded? Thanks in advance.

0 Karma

Builder

Anything in your splunkd.log file that might indicate the source of a problem?

Does the logging seem to happen for a particular length of time or a certain amount of data before out to lunch (i.e. every 7 days or every 1GB of data)?

0 Karma

Explorer

The only think in the splunkd log at the time is a "WARN DateParserVerbose - Failed to parse timestamp for event"...

The source shows a number of events with timestamps and then two lines at the end with no timestamps. The very last line also contains the escaped comma.

Is it a possible glitch that events with no timestamp and an escaped comma can cause splunk to stop indexing? What can I do to test or work around?

0 Karma

Explorer

More info on this. Looks like there is a common last entry before the indexing stops. "Copyright (c) 1986-2010 by Cisco Systems\, Inc." is the last entry indexed before it stops and has to be restarted. Is that backslash (which I assume is escaping the comma) potentially causing splunk indexing to stop? Is it something with that last entry or is it perhaps the next entry that might be funky and causes splunk indexing to stop before it gets recorded?

0 Karma

Explorer

I have also learned that just stopping and starting splunkd gets the indexing going again. Splunkweb continues to run fine, allows searches etc...

0 Karma

Builder

We have the same issues, but out system is severly overloaded.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!