Deployment Architecture

Splunk stops logging, requires service restart

cbdick
Explorer

We have splunk logging via inbound syslog stream. Recently, we have found that Splunk is no longer logging the data (doing a wildcard search in realtime, no entries appear). We restart the service (splunk stop/start) and that seems to kick it back into gear. This is starting to happen more and more frequently. Any ideas where to look to determine why this is happening and how to resolve?

We are running splunk 4.1.6 on 64 bit windows server.

0 Karma
1 Solution

cbdick
Explorer

The issue was not that Splunk stopped indexing. It turned out to be that the timestamp of the current time changed to a year in the past. I will close this post and open another to try to determine why the current timestamp changed and how to workaround with props.conf.

View solution in original post

0 Karma

cbdick
Explorer

The issue was not that Splunk stopped indexing. It turned out to be that the timestamp of the current time changed to a year in the past. I will close this post and open another to try to determine why the current timestamp changed and how to workaround with props.conf.

0 Karma

cbdick
Explorer

We have determined that whatever is causing indexing to stop/crash is likely due to the next message coming in via syslog port. This message does not actually get written to the index and it does not seem to trigger anything in the splunkd.log. Is there any other locations I can look to see if the contents of this message are getting recorded? Thanks in advance.

0 Karma

mfrost8
Builder

Anything in your splunkd.log file that might indicate the source of a problem?

Does the logging seem to happen for a particular length of time or a certain amount of data before out to lunch (i.e. every 7 days or every 1GB of data)?

0 Karma

cbdick
Explorer

The only think in the splunkd log at the time is a "WARN DateParserVerbose - Failed to parse timestamp for event"...

The source shows a number of events with timestamps and then two lines at the end with no timestamps. The very last line also contains the escaped comma.

Is it a possible glitch that events with no timestamp and an escaped comma can cause splunk to stop indexing? What can I do to test or work around?

0 Karma

cbdick
Explorer

More info on this. Looks like there is a common last entry before the indexing stops. "Copyright (c) 1986-2010 by Cisco Systems\, Inc." is the last entry indexed before it stops and has to be restarted. Is that backslash (which I assume is escaping the comma) potentially causing splunk indexing to stop? Is it something with that last entry or is it perhaps the next entry that might be funky and causes splunk indexing to stop before it gets recorded?

0 Karma

cbdick
Explorer

I have also learned that just stopping and starting splunkd gets the indexing going again. Splunkweb continues to run fine, allows searches etc...

0 Karma

fk319
Builder

We have the same issues, but out system is severly overloaded.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...