Hi all,
Hopefully someone can assist me here. We are using Splunk Light Version 6.2.3 but have discovered recently that Splunk seems to stop logging for a few days once a new month starts.
For example, here is an extract of two random months this year:
April 30th 2017 - 123,323 Events
May 1st 2017 - 388 Events
May 2nd 2017 - 0 Events
May 3rd 2017 - 0 Events
May 4th 2017 - 0 Events
July 31st 2017 - 281,966 Events
August 1st 2017 - 426 Events
August 2nd 2017 - 0 Events
August 3rd 2017 - 0 Events
August 4th 2017 - 0 Events
August 5th 2017 -0 Events
August 6th 2017 - 0 Events
August 7th 2017 - 0 Events
August 8th 2017 - 327,876 Events
The same scenario has happened throughout the time we have been using Splunk, but we have only just spotted this today after looking at a yearly view.
Has anyone seen this issue before? Can anyone recommend a few troubleshooting tips?
Thanks in advance,
Jonathan
This looks like a date parsing problem. I'm guessing the raw data has dates in dd/mm/yyyy format, but Splunk is trying to read them as mm/dd/yyyy/ format. You can confirm this by looking at the events on 8 Feb 17 (2/8/17 US) to see if there some that should be dated 2 Aug 17 (2/8/17 RoW).
If you confirm this is what is happening then the fix is simple. Modify your props.conf file to include a TIME_FORMAT=
attribute for the appropriate sourcetype(s).
Hi JonzOo,
probably the problem is that there's a wrong configuration of Timestamp.
In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).
So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).
Verify your TIME_FORMAT or share an example of your log.
Bye.
Giuseppe
Have same issue. Can you please clarify what changes do i need for these lines:
[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
HI @JonzOo,
as I said probably the problem is that there's a wrong configuration of Timestamp.
In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).
So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).
Verify your TIME_FORMAT or share an example of your log.
If you want an help in this check you should share some log example to check your TIME_FORMAT.
I hint to open new question so it will possible for you to accept the answer.
Ciao.
Giuseppe
Hi cusello,
Thank you. After going back to look at the results, you are correct.
I will have a look at editting the props.conf file to add the TIME_FORMAT into it.
Thanks,
Jonathan
If you're satisfied by this answer, please accept or upvote it.
Thank you.
Bye.
Giuseppe
This looks like a date parsing problem. I'm guessing the raw data has dates in dd/mm/yyyy format, but Splunk is trying to read them as mm/dd/yyyy/ format. You can confirm this by looking at the events on 8 Feb 17 (2/8/17 US) to see if there some that should be dated 2 Aug 17 (2/8/17 RoW).
If you confirm this is what is happening then the fix is simple. Modify your props.conf file to include a TIME_FORMAT=
attribute for the appropriate sourcetype(s).
Hi Rich,
You're spot on with that answer. I can now see the pattern with the dates.
I've never dealt with the configuration of Splunk so i'll have a look into it and see what I can do.
Thank you very much 🙂
Jonathan