Getting Data In

My data stops logging at the beginning of the month

Explorer

Hi all,

Hopefully someone can assist me here. We are using Splunk Light Version 6.2.3 but have discovered recently that Splunk seems to stop logging for a few days once a new month starts.

For example, here is an extract of two random months this year:

April 30th 2017 - 123,323 Events
May 1st 2017 - 388 Events
May 2nd 2017 - 0 Events
May 3rd 2017 - 0 Events
May 4th 2017 - 0 Events

May 5th 2017 - 287,234 Events

July 31st 2017 - 281,966 Events
August 1st 2017 - 426 Events
August 2nd 2017 - 0 Events
August 3rd 2017 - 0 Events
August 4th 2017 - 0 Events
August 5th 2017 -0 Events
August 6th 2017 - 0 Events
August 7th 2017 - 0 Events
August 8th 2017 - 327,876 Events

The same scenario has happened throughout the time we have been using Splunk, but we have only just spotted this today after looking at a yearly view.

Has anyone seen this issue before? Can anyone recommend a few troubleshooting tips?

Thanks in advance,
Jonathan

0 Karma
1 Solution

SplunkTrust
SplunkTrust

This looks like a date parsing problem. I'm guessing the raw data has dates in dd/mm/yyyy format, but Splunk is trying to read them as mm/dd/yyyy/ format. You can confirm this by looking at the events on 8 Feb 17 (2/8/17 US) to see if there some that should be dated 2 Aug 17 (2/8/17 RoW).

If you confirm this is what is happening then the fix is simple. Modify your props.conf file to include a TIME_FORMAT= attribute for the appropriate sourcetype(s).

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Legend

Hi JonzOo,
probably the problem is that there's a wrong configuration of Timestamp.
In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).
So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).
Verify your TIME_FORMAT or share an example of your log.
Bye.
Giuseppe

Explorer

Hi cusello,

Thank you. After going back to look at the results, you are correct.

I will have a look at editting the props.conf file to add the TIME_FORMAT into it.

Thanks,
Jonathan

0 Karma

Legend

If you're satisfied by this answer, please accept or upvote it.
Thank you.
Bye.
Giuseppe

0 Karma

SplunkTrust
SplunkTrust

This looks like a date parsing problem. I'm guessing the raw data has dates in dd/mm/yyyy format, but Splunk is trying to read them as mm/dd/yyyy/ format. You can confirm this by looking at the events on 8 Feb 17 (2/8/17 US) to see if there some that should be dated 2 Aug 17 (2/8/17 RoW).

If you confirm this is what is happening then the fix is simple. Modify your props.conf file to include a TIME_FORMAT= attribute for the appropriate sourcetype(s).

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Explorer

Hi Rich,

You're spot on with that answer. I can now see the pattern with the dates.

I've never dealt with the configuration of Splunk so i'll have a look into it and see what I can do.

Thank you very much 🙂
Jonathan

0 Karma