Log File Monitoring giving me the future timestamp

[sarvesh_11]

Hello folks,
Would like to grab your intention, on my current issue with Splunk. Please help me with you r valuable inputs.
I am monitoring logs file in splunk. Which has foretasted computation period.

Example Computation period is from 00:00:00 to 23:59:59 27th March,2019.

So my batch is running every hour, and logging it to a directory in a log file.
Now when i look for these logs on splunk, i gets 3 different time stamp.

1 is the 12:00:00 AM 27th March, 2019.
2 is my current hour, on which i my cron is running.
3 is 23:59:59 27th March, 2019.

These are the time stamp for each of the cron.

What configuration changes shall i make, to make it a 1 single time stamp, i.e the current cron time?
Well this is scenario of 1 of my batch, i have 3 more, which has computation period of a week also.

[solarboyz1 · Accepted solution]

Splunk assigns timestamps as follows:
https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/HowSplunkextractstimestamps

Splunk software uses the following precedence rules to assign timestamps to events:

1. It looks for a time or date in the event itself using an explicit TIME_FORMAT, if provided. You configure the TIME_FORMAT attribute in props.conf.

2. If no TIME_FORMAT was configured for the data, Splunk software attempts to automatically identify a time or date in the event itself. It uses the source type of the event (which includes TIME_FORMAT information) to try to find the timestamp.

3. If an event has a time and date, but not a year, Splunk software determines the year, as described in How Splunk software determines timestamps with no year, and builds the timestamp from that.

4. If no events in a source have a date, Splunk software tries to find a date in the source name or file name. Time of day is not identified in filenames. (This requires that the events have a time, even though they don't have a date.)

5. For file sources, if no date can be identified in the file name, Splunk software uses the file modification time.

6. As a last resort, Splunk software sets the timestamp to the current system time when indexing each event.

If you want to configure it so all events in the file receive the same timestamp, you can set

DATETIME_CONFIG = NONE

This should set the timestamp for all events to the modification time of the file:
https://docs.splunk.com/Documentation/Splunk/7.2.5/admin/Propsconf#Timestamp_extraction_configuratio...

For file-based inputs (monitor, batch) the time chosen will be the modification timestamp on the file being read.

Which should be the end of the cron execution, when the last event was written to the file.

[sarvesh_11]

Hi @solarboyz1 ,
Thanks for replying.

1st) I was trying to make changes on /opt/SplunkUniversalForwarder/etc/system/local/, here i have created one props.conf, evaluating my sourcetype, with DATETIME_CONFIG=NONE

2nd) I cannot use the timestamp from the events because, the 1st issue what m facing is, i am not getting the data 1 file in single shot on search head, as it is taking the time stamp from the event. so it is showing forecasted time also, where as i just want to monitor this file every hour. i cannot increase my time range to past 24 hours. and also, the conclusion sentence , the keywords we wants to capture, appears at the last of every file, which is visible at 23:59:59, like if i am monitoring file at 9AM i wont be able to see that conclusion sentence in that file at that time.

3rd) Are you expecting me to make changes in the props.conf of Indexer, till now what changes i am making was all in Universal Forwarder.

SO what i am finally left is the index time of 1 file should be sync with the content of that file. i.r 1 source file should have only 1 timestamp. i.e the last modified file time.

[solarboyz1]

the only way to do this on the search head via SPL, would be to make the timestamp you want to use available in the event.

For example if the files were created with the timestamp in the name: logfile-03_04_2005_01_02_30.log
then you could do something like:

| rex field=source ".*-(?<mon>\d+)_(?<day>\d+)_(?<year>\d+)_(?<hour>\d+)_(?<min>\d+)_(?<sec>\d+).log" 
| eval event_time=mon. "/" .day. "/" .year. ":" .hour. ":" .min. "sec"
| eval _time=strptime(event_time, "%m/%d/%Y:%H:%M:%S")
| timechart count 

[sarvesh_11]

Hi @solarboyz1 ,
can this be handled by outputs.conf?
if i define below in system/local/outputs.conf

[sourcetype / source]
DATETIME_CONFIG = none

[solarboyz1]

No.

outputs.conf deals with how the data is sent to other instances (i.e. UF to indexers). The options you listed are not valid for that configuration file:

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf

[sarvesh_11]

Thanks Man @solarboyz1
Appreciate your help.

Will do changes in props.conf itself for time stamp resolution.
[sourcetype]
DATETIME_CONFIG= none

[sarvesh_11]

Hey @solarboyz1
Really appreciate your prompt response.
This can be handle on the search head, i agree!
But as i mentioned, the scenario is,
Lets take an example, my log file is created at 02:00:00 AM at my source, now when my cron runs, i get the logs on splunk at 2:10:10. 3rd April But not the complete logs, not the complete content of the file is visible.
Below content i am able to see at at 23:59:59, while this is written already in the log file, which is modified at 02:00:00

"Bill End Date : 04/03/2019 23:59:59
Scheduled Read Date: 04/03/2019
Number of Days : 1
-------------------------------- Page ----------------------------------
--- RunRS completed successfully ---"

Because of this ambiguity, i am focusing only on to get single time stamp for 1 log file. i am giving DATETIME_CONFIG= none in props.conf (/opt/SplunkUniversalForwarder/system/local/). Not sure i shall get the result with this or not.

[solarboyz1]

Are you expecting me to make changes in the props.conf of Indexer, till now what changes i am making was all in Universal Forwarder

I think the timestamp and hostname are extracted on the UF, so it would probably work if only on the UF. But Im not positive which is why I recommended putting it on the indexer as well.

[sarvesh_11]

Hi @solarboyz1 ,
Can you propose any other solution for the same. Can we control this on Search Head SPL Query.

[somesoni2]

Could you please share some sample raw data from your logs and what's the current event process/sourcetype config/props.conf entry you've setup?

[sarvesh_11]

Hi @somesoni2 ,
Thanks for replying.

1st) I was trying to make changes on /opt/SplunkUniversalForwarder/etc/system/local/, here i have created one props.conf, evaluating my sourcetype, with DATETIME_CONFIG=NONE

2nd) I cannot use the timestamp from the events because, the 1st issue what m facing is, i am not getting the data 1 file in single shot on search head, as it is taking the time stamp from the event. so it is showing forecasted time also, where as i just want to monitor this file every hour. i cannot increase my time range to past 24 hours. and also, the conclusion sentence , the keywords we wants to capture, appears at the last of every file, which is visible at 23:59:59, like if i am monitoring file at 9AM i wont be able to see that conclusion sentence in that file at that time.

3rd) Are you expecting me to make changes in the props.conf of Indexer, till now what changes i am making was all in Universal Forwarder.

SO what i am finally left is the index time of 1 file should be sync with the content of that file. i.r 1 source file should have only 1 timestamp. i.e the last modified file time.

[sarvesh_11]

Hi Somesh,
I guess no, i am not havnig enough karma points to put ant screenshot or file.
Although in props.sonf i have defined DATETIME_CONFIG=auto.
I just want to clarify, if splunk is taking the time stamp from the event inside the log file, instead of the timestamp of the file modified. How can i get the sinlge time stamp, i.e ignore or manipulate the event time stamp, what it is taking from inside the log file.

[solarboyz1]

Hi Sarvesh_11,

As described in my answer to your question, you need to set your props.conf to:

DATETIME_CONFIG = NONE

As I explain below, this would give all events the same timestamp, based on the modification time.