topic Re: Splunk stop to process syslog messages every 7 days in Getting Data In

Splunk stop to process syslog messages every 7 days

petpet — Mon, 13 Jan 2014 10:50:21 GMT

Hi
i noticed that every seven days at 4:03 ( of the local time )splunk stop to process Syslog messages. then i need to restart the splunk and it start again.
here is the excerpt log of the splunkd.log, when it stopped and when i restart it:

1-11-2014 23:00:06.249 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 23:00:05 2014). Context: source::Syslog|host::10.255.196.2|syslog|
01-11-2014 23:04:06.251 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 23:04:05 2014). Context: source::Syslog|host::10.255.196.2|syslog|
01-11-2014 23:23:17.335 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 19:23:16 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-11-2014 23:28:28.082 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 19:28:26 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-12-2014 00:00:00.984 +0100 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1389394800 lastRolloverDay=1389394800 snappedNow=1389481200
01-12-2014 00:00:00.985 +0100 INFO LMStackMgr - quotaExceededCount=0, lastExceedDate=0, peak=23676253, rolloverCount=6, totalCumulativeBytesAtRollover=23676253, todaysBytesIndexed=23676253, licenseSize=524288000
01-12-2014 00:00:00.985 +0100 INFO LMStackMgr - finished rollover, new lastRolloverTime=1389481200
01-12-2014 00:00:41.985 +0100 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Sat Jan 11 23:59:41 2014) < lastRolloverTime(Sun Jan 12 00:00:00 2014), meaning that the master has already rolled over. Ignore slave persisted usage.
01-12-2014 00:12:43.985 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 20:12:42 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-12-2014 00:14:08.058 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 20:14:06 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-12-2014 01:07:35.661 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/splunk/metrics.log'.
01-12-2014 01:07:35.702 +0100 INFO WatchedFile - Will begin reading at offset=24996941 for file='/opt/splunk/var/log/splunk/metrics.log.1'.
01-12-2014 02:08:48.264 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 22:08:47 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-12-2014 02:12:06.816 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 22:12:05 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-12-2014 02:13:14.933 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 22:13:13 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-12-2014 02:21:28.954 +0100 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jan 11 22:21:27 2014). Context: source::Syslog|host::10.27.1.3|syslog|
01-12-2014 22:36:40.895 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/splunk/metrics.log'.
01-12-2014 22:36:40.897 +0100 INFO WatchedFile - Will begin reading at offset=24992775 for file='/opt/splunk/var/log/splunk/metrics.log.1'.
01-13-2014 00:00:00.984 +0100 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1389481200 lastRolloverDay=1389481200 snappedNow=1389567600
01-13-2014 00:00:00.985 +0100 INFO LMStackMgr - quotaExceededCount=0, lastExceedDate=0, peak=4747595, rolloverCount=7, totalCumulativeBytesAtRollover=4747595, todaysBytesIndexed=4747595, licenseSize=524288000

Many thanks for any advice

Peter

Re: Splunk stop to process syslog messages every 7 days

Ayn — Mon, 13 Jan 2014 11:50:55 GMT

Those DateParser messages don't look good. Improper timestamp parsing can cause various problems. My advice is to fix that first of all before doing any further troubleshooting.

Re: Splunk stop to process syslog messages every 7 days

petpet — Mon, 13 Jan 2014 13:49:43 GMT

thank you for the answer
what does it mean to fix the timestamp parsing ?
I'm collecting the syslog messages from the routers from around the world from different timezones
I'm using newest default Splunk installation, just created new data source name Syslog

thanks

Re: Splunk stop to process syslog messages every 7 days

yannK — Mon, 13 Jan 2014 19:20:06 GMT

To check the timestamp parsing.

Export a sample of the events in a file on the search-head, and index them using the data preview. Apply the syslog sourcetype, verify the timestamp extraction.

FYI, syslog events are supposed to be single line events, starting with a timestamp, then the host, then the events.

Verify that your data is actual syslog format (check that the host present and correctly extracted from the events.
If your events are not all syslog, use a different sourcetype.
The best solution is to create multiple listening ports, one for each format, and redirect your devices to the correct port.

Re: Splunk stop to process syslog messages every 7 days

petpet — Mon, 28 Sep 2020 15:39:01 GMT

thanks for your answer. for example, this is what was sent by the router, as the one event:

Jan 14 10:26:21: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed
connection id=489, sequence number=1252

And processed by the Splunk as the 3 events:
_raw
<140>2018:
<140>2017: connection id=489, sequence number=1252"
<140>2016: Jan 14 10:26:21: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed

So seems that router itself send wrong format of the syslog message.
How can i protect it on the Splunk side, to either ignore it or process it ?

Thanks

Re: Splunk stop to process syslog messages every 7 days

petpet — Mon, 10 Feb 2014 09:23:52 GMT

Hi
actually i was not able to fix the wrong messages from the Cisco routers, however this is still strange why the Splunk stops to process Syslog events every 7 days at 4:03 AM.
Of course there is a solution to set the corn to restart the splunk at 4:05 AM, however i think there must be something more than the parsers
thanks for any hint