Splunk Search

splunk _time not matching with timestamp

nawazns5038
Builder

Hi,

the log has timestamp like this "time":"2018-01-22 13:43:40.0"

props.conf :
TIME_FORMAT = %F %T.%3N
TIME_PREFIX = "time":\"
MAX_TIMESTAMP_LOOKAHEAD = 25

that was the props conf used. The setting worked fine while testing. After indexing the data the timestamp shown in Splunk or _time has come upto "11/28/17 4:06:53.568 PM" . which is even no where present in the event.

How can this be resolved. Please help.

0 Karma

nickhills
Ultra Champion

Are the events 'breaking' properly? Ie, one valid json block per event?
Also is it aws generated data (s3/cloudwatch logs) etc, or your own log data?

If my comment helps, please give it a thumbs up!
0 Karma

nawazns5038
Builder

Ya breaking is proper and props has been tested with the sample data as well. It is only one Json block.
Data is pulled from S3 buckets and it is not AWS default data . and it is pulling lots of .gz files which has json files in it
Is it a problem related to the Addon ?

0 Karma

micahkemp
Champion

Can you paste the full _raw of the event in question? Sometimes there are multiple instances of timestamps on one message, which can confuse Splunk.

0 Karma

nawazns5038
Builder

@p_gaurav

There is only time field and all the events are of the same format...Some are taking the correct value and some are not even in the range or just picking up some random timestamp which is not even present in the event
The data is being sent from S3 input from AWS addon

0 Karma

p_gurav
Champion

Can you check _internal logs for particular S3 input? Is there any timestamp related warning or error?

0 Karma

nawazns5038
Builder

It's really strange ... "time":"2018-01-21 05:42:34.0"
There is only time field and all the events are of the same format...Some are taking the correct value and some are not even in the range or just picking up some random timestamp which is not even present in the event
The data is being sent from S3 input from AWS addon

0 Karma

p_gurav
Champion

Hi nawazns5038,
Could you please provide sample events? Also in TIME_PREFIX try changing rex to \"time\"\:\"

0 Karma
Get Updates on the Splunk Community!

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...