one input stopped indexing exactly at midnight when starting a new month

[sander_vandamme]

We saw that one input has stopped indexing exactly at midnight when starting a new month. So we have the correct data untill 28/02/2018 23:59:59 and after midnight no data is visible anymore.
The Splunk Forwarder is working correctly and is sending all the data from all other inputs without any issues. The forwarder is also not reporting any problems with indexing the inputs that has stopped working.

I tried restarting the forwarder, restarting the indexer (all in one Splunk Enterprise) and I alsready tried uninstalling the app on this server that holds this input (using the Splunk Deployment server).

The forwarder does recognize the new files in the input directory just fine but it seems like it is not indexed correctly as we cannot search the data.

How can I further troubleshoot what happened or is this a known bug in this version of Splunk?

Splunk Enterprise:
VERSION=6.6.3
BUILD=e21ee54bc796
PRODUCT=splunk
PLATFORM=Linux-x86_64
Splunk Universal Forwarder:
VERSION=6.6.3
BUILD=e21ee54bc796
PRODUCT=splunk
PLATFORM=Linux-x86_64

[gcusello · Accepted solution]

Hi sander_vandamme1,
the problem is that you have dates in format dd/mm/yyy instead Splunk by default read mm/dd/yyyy, so:

• if you have 28/02/2018, data is correctly read,
• if you have 01/03/2018 (1st of march) data is read in wrong way.

If you search logs of the 1st of march using as search date 3rd of january, you'll find your events.

To solve the problem, you have to correctly set your DATE_FORMAT in props.conf for the sourcetypes you're ingesting.

[my_sourcetype]
DATE_FORMAT = %d/%m/%Y %H:%M:%S

Bye.
Giuseppe

[sander_vandamme]

That was the problem! Thank you so much!