topic Re: Why is Splunk indexing data for 12 hours instead of 24 hours? in Getting Data In

Why is Splunk indexing data for 12 hours instead of 24 hours?

gautamr103 — Wed, 14 Feb 2018 18:49:08 GMT

After 12:59 PM slpunk is indexing data to 1:AM. It should index data for 24 hours but it is indexing for 12 hours only however 1:PM data are getting indexed in 1Am so I have two events in 1 am time stamp Below is my props.conf file.

DATETIME_CONFIG=CURRENT
NO_BINARY_CHECK = 1
pulldown_type = 1
TIME_FORMAT = %H:%M
TZ = US/Eastern
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = \d\d:\d\d+\s*$
MAX_TIMESTAMP_LOOKAHEAD = 50

Re: Why is Splunk indexing data for 12 hours instead of 24 hours?

micahkemp — Wed, 14 Feb 2018 20:21:33 GMT

What does your event text look like? If it includes AM/PM your TIME_FORMAT won't handle that.

If your event looks like:

03:45 PM

Your TIME_FORMAT would need to be:

TIME_FORMAT = %H:%M %p

Re: Why is Splunk indexing data for 12 hours instead of 24 hours?

gautamr103 — Tue, 29 Sep 2020 18:07:16 GMT

Hi Micahkemp thank you for your reply

I dont have am pm on my event logs

this is my logs generated at 12:01 AM---> 12:01 Info [tasks_advancemedia_aspx]
and this was generated at 12:01 PM --> 12:01 Info [WorkerService] RTAEncode acm status

Also I wanted to break event according to time I have another log at same time ---> 12:01 Error [lambda_method] Unable how would I break event with same time but different logs, I tried BREAK_ONLY_BEFORE = ^\d\d:\d\d+\s but it did not work.

Re: Why is Splunk indexing data for 12 hours instead of 24 hours?

micahkemp — Wed, 14 Feb 2018 22:01:43 GMT

So your events look like:

12:01     <-- 12:01 AM
01:01     <-- 1:01 AM
...
12:01    <-- 12:01 PM
01:01    <-- 1:01 PM

Which would mean you don't have AM/PM or 24-hour format. That sounds less than ideal to say the least.

Re: Why is Splunk indexing data for 12 hours instead of 24 hours?

gautamr103 — Wed, 14 Feb 2018 22:10:24 GMT

right without AM and PM

Re: Why is Splunk indexing data for 12 hours instead of 24 hours?

thomast_splunk — Wed, 16 Jan 2019 22:46:33 GMT

May want to just use index time if in the same timezone - or keep that in mind for this particular sourcetype

Props.conf

[sourcetypeName]
DATETIME_CONFIG = CURRENT

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf?utm_source=answers&utm_medium=in-comment&utm_term=props.conf&utm_campaign=refdoc#Timestamp_extraction_configuration

Re: Why is Splunk indexing data for 12 hours instead of 24 hours?

eavent_splunk — Thu, 17 Jan 2019 05:30:00 GMT

Just to expand on previous comments - it is indexing for 24 hours, but the lack of AM/PM data is resulting in everything being in AM.

If you source data cannot be adjusted to include more time information, then as @thomast_splunk suggests one option would be to just use the whatever the current time and date is when splunk receives the event for processing.

DATETIME_CONFIG = NONE is another option:

  * "NONE" will leave the event time set to whatever time was selected by
    the input layer
    * For data sent by splunk forwarders over the splunk protocol, the input
      layer will be the time that was selected on the forwarder by its input
      behavior (as below).
    * For file-based inputs (monitor, batch) the time chosen will be the
      modification timestamp on the file being read.
    * For other inputs, the time chosen will be the current system time when
      the event is read from the pipe/socket/etc.

This page is a good primer on how Splunk assigns timestamps if you want more details:
https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/HowSplunkextractstimestamps#How_Splunk_software_assigns_timestamps

Re: Why is Splunk indexing data for 12 hours instead of 24 hours?

nikita_p — Tue, 29 Sep 2020 22:47:24 GMT

Hi,
Can you give some sample of your time format. So if you also have seconds and milliseconds in your events, then you will have to change your TIME_FORMAT in props.conf as below:
TIME_FORMAT=%H:%M:%S.%N