Getting Data In

My data stops logging at the beginning of the month

JonzOo
Explorer

Hi all,

Hopefully someone can assist me here. We are using Splunk Light Version 6.2.3 but have discovered recently that Splunk seems to stop logging for a few days once a new month starts.

For example, here is an extract of two random months this year:

April 30th 2017 - 123,323 Events
May 1st 2017 - 388 Events
May 2nd 2017 - 0 Events
May 3rd 2017 - 0 Events
May 4th 2017 - 0 Events

May 5th 2017 - 287,234 Events

July 31st 2017 - 281,966 Events
August 1st 2017 - 426 Events
August 2nd 2017 - 0 Events
August 3rd 2017 - 0 Events
August 4th 2017 - 0 Events
August 5th 2017 -0 Events
August 6th 2017 - 0 Events
August 7th 2017 - 0 Events
August 8th 2017 - 327,876 Events

The same scenario has happened throughout the time we have been using Splunk, but we have only just spotted this today after looking at a yearly view.

Has anyone seen this issue before? Can anyone recommend a few troubleshooting tips?

Thanks in advance,
Jonathan

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

This looks like a date parsing problem. I'm guessing the raw data has dates in dd/mm/yyyy format, but Splunk is trying to read them as mm/dd/yyyy/ format. You can confirm this by looking at the events on 8 Feb 17 (2/8/17 US) to see if there some that should be dated 2 Aug 17 (2/8/17 RoW).

If you confirm this is what is happening then the fix is simple. Modify your props.conf file to include a TIME_FORMAT= attribute for the appropriate sourcetype(s).

---
If this reply helps you, Karma would be appreciated.

View solution in original post

gcusello
Esteemed Legend

Hi JonzOo,
probably the problem is that there's a wrong configuration of Timestamp.
In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).
So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).
Verify your TIME_FORMAT or share an example of your log.
Bye.
Giuseppe

dunyaelbasan
Path Finder

Have same issue. Can you please clarify what changes do i need for these lines:

 

[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF

0 Karma

gcusello
Esteemed Legend

HI @JonzOo,

as I said probably the problem is that there's a wrong configuration of Timestamp.
In other words you probably have in your logs a date in European format (dd/mm/yyyy hh.mm.ss), instead Splunk read it in American format (mm/dd/yyyy hh:mm:ss), infact Splunk correctly read your timestamp when day and month are the same or when there's non dubt (e.g. days greater than 12).
So your logs are indexed with a wrong date (e.g. 1st of September is read as 9th of January).
Verify your TIME_FORMAT or share an example of your log.

If you want an help in this check you should share some log example to check your TIME_FORMAT.

I hint to open  new question so it will possible for you to accept the answer.

Ciao.

Giuseppe

0 Karma

JonzOo
Explorer

Hi cusello,

Thank you. After going back to look at the results, you are correct.

I will have a look at editting the props.conf file to add the TIME_FORMAT into it.

Thanks,
Jonathan

0 Karma

gcusello
Esteemed Legend

If you're satisfied by this answer, please accept or upvote it.
Thank you.
Bye.
Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This looks like a date parsing problem. I'm guessing the raw data has dates in dd/mm/yyyy format, but Splunk is trying to read them as mm/dd/yyyy/ format. You can confirm this by looking at the events on 8 Feb 17 (2/8/17 US) to see if there some that should be dated 2 Aug 17 (2/8/17 RoW).

If you confirm this is what is happening then the fix is simple. Modify your props.conf file to include a TIME_FORMAT= attribute for the appropriate sourcetype(s).

---
If this reply helps you, Karma would be appreciated.

JonzOo
Explorer

Hi Rich,

You're spot on with that answer. I can now see the pattern with the dates.

I've never dealt with the configuration of Splunk so i'll have a look into it and see what I can do.

Thank you very much 🙂
Jonathan

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...