My data stops logging at the beginning of the month

[JonzOo]

Hi all,

Hopefully someone can assist me here. We are using Splunk Light Version 6.2.3 but have discovered recently that Splunk seems to stop logging for a few days once a new month starts.

For example, here is an extract of two random months this year:

April 30th 2017 - 123,323 Events
May 1st 2017 - 388 Events
May 2nd 2017 - 0 Events
May 3rd 2017 - 0 Events
May 4th 2017 - 0 Events

May 5th 2017 - 287,234 Events

July 31st 2017 - 281,966 Events
August 1st 2017 - 426 Events
August 2nd 2017 - 0 Events
August 3rd 2017 - 0 Events
August 4th 2017 - 0 Events
August 5th 2017 -0 Events
August 6th 2017 - 0 Events
August 7th 2017 - 0 Events
August 8th 2017 - 327,876 Events

The same scenario has happened throughout the time we have been using Splunk, but we have only just spotted this today after looking at a yearly view.

Has anyone seen this issue before? Can anyone recommend a few troubleshooting tips?

Thanks in advance,
Jonathan

[gcusello]

Hi JonzOo,
probably the problem is that there's a wrong configuration of Timestamp.
In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).
So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).
Verify your TIME_FORMAT or share an example of your log.
Bye.
Giuseppe

[dunyaelbasan]

Have same issue. Can you please clarify what changes do i need for these lines:

[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF

[gcusello]

HI @JonzOo,

as I said probably the problem is that there's a wrong configuration of Timestamp.
In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).
So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).
Verify your TIME_FORMAT or share an example of your log.

If you want an help in this check you should share some log example to check your TIME_FORMAT.

I hint to open  new question so it will possible for you to accept the answer.

Ciao.

Giuseppe

[JonzOo]

Hi cusello,

Thank you. After going back to look at the results, you are correct.

I will have a look at editting the props.conf file to add the TIME_FORMAT into it.

Thanks,
Jonathan

[gcusello]

If you're satisfied by this answer, please accept or upvote it.
Thank you.
Bye.
Giuseppe

[richgalloway · Accepted solution]

This looks like a date parsing problem. I'm guessing the raw data has dates in dd/mm/yyyy format, but Splunk is trying to read them as mm/dd/yyyy/ format. You can confirm this by looking at the events on 8 Feb 17 (2/8/17 US) to see if there some that should be dated 2 Aug 17 (2/8/17 RoW).

If you confirm this is what is happening then the fix is simple. Modify your props.conf file to include a TIME_FORMAT= attribute for the appropriate sourcetype(s).

[JonzOo]

Hi Rich,

You're spot on with that answer. I can now see the pattern with the dates.

I've never dealt with the configuration of Splunk so i'll have a look into it and see what I can do.

Thank you very much 🙂
Jonathan