Hi guys,
I'm having trouble in getting the right timestamp from my log file.
Please refer to this image .. http://postimg.org/image/mvzxzyfzv/
Splunk is getting 4/16/2015
as the _time
, but what i want is to get the first field in the example which is 04/20/2015 HH:MM:SS
I already tried using some regex like ^\"\d+\/\d+\/\d_\s\d+\:\d+\:\d+
to get the timestamp, but it's failing to get the right time.
Please help me guys!
Thanks!
Hi,
Your issue is that your events are more than 2 days in the future.. splunk will ignore timestamps for more than 2 days in future.
You can use the MAX_DAYS_HENCE Parameter in props.conf go get it working.
Regards,
Andreas
Try this:
........|rex "\"(?<date>\d+\/\d+\/\d+\s\d+\:\d+\:\d+)\. "|table date
@stephanefotso, this may work also, but I need to put the regex on the props.conf so that before indexing the data,it will capture the field that i want to be the time stamp of my event.
Hi,
Your issue is that your events are more than 2 days in the future.. splunk will ignore timestamps for more than 2 days in future.
You can use the MAX_DAYS_HENCE Parameter in props.conf go get it working.
Regards,
Andreas
hello @schose, this may work but the scenario is i have a log file consists of data from Mar 11 2015 up until today. So i need to index all of those. from my previous config in props.conf I successfully capture the right timestamp from dates Mar 11 to April 16, when splunk see's the log from Mar 17, it gets the timestamp from another field, not on the field that I needed.
@schose maybe i can use MAX_TIMESTAMP_LOOKAHEAD right? and set it to 12 or 41? what do you think ?
Hey your screenshot it showing a date from 20th of april... can you use a older logfile, reproduce and take a screenshot of the issue - cause I'm sure it's the issue - i tested it out.
Regards,
hi @schose here's the screen shots:
http://postimg.org/image/n57egmbmb/
http://postimg.org/image/b2qp8zbif/
its just a sample, i've got a lot of those events that didnt get my desired timestamp
my desired timestamp is the one enclosed in the green rectangle
my props.conf
[download_logs_ftp_two]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\d+\,\"
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
No. If splunk is told that a timestamp can be no further in the future than two days, it will not accept a timestamp from further in the future even if this is the only timestamp there is in an event. As schose said, you need to set MAX_DAYS_HENCE to a higher value; it shouldn't harm your system to incease it a few days.
hello @jeffaland wat if i use MAX_TIMESTAMP_LOOKAHEAD ? i cannot use the MAX_DAYS_HENCE right now because i have logs from MAr 2015. Am I right?
Well there's two things you need to do:
a) you need to show splunk precisely which timestamp to use if there is more than one (such as in your screenshot) and
b) if (from todays perspective) your timestamp is supposed to be the 21st of April (also like in your screenshot), you need to allow those future timestamps.
from my props.conf, i already put TIME_PREFIX so that splunk would capture the timestamp i needed. It works on majority of the events, but i have a problem with some of the events because it's not getting the timestamp i defined with the TIME_PREFIX
[download_logs_ftp_two]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\d+\,\"
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
that's why I'm thinking of using MAX_TIMESTAMP_LOOKAHEAD so that splunk will look for timestamps from the defined integer, for like example default value is 150 so it looks like MAX_TIMESTAMP_LOOKAHEAD=150
so it would get characters in my event from 1-150. Is it right?
Generally speaking, yes.
what about the second thing i need to do? please elaborate ..
The green rectangle you mentioned which represents your desired timestamp, it shows a date which is in the future. By default, splunk will only accept timestamps which are up to two days in the future; yours is three. So, as schose said, you'll need to set MAX_DAYS_HENCE to a higher value. This is done in props.conf, see here.
so all in all i have to put in my props.conf the ff:
MAX_DAYS_HENCE=4
MAX_TIMESTAMP_LOOKAHEAD=29
TIME_PREFIX=^\d+\,\"
is it right?
You should try it and be able to see if it works 🙂 but that's what I think is needed.
If some of your dates are even further in the future, setting MAX_DAYS_HENCE to the required amount might be needed as well.
Thanks! will try this 🙂 hope it works .. 🙂
@jeffland in my logs, you can see lots of dates and time, so i have to specify the field where splunk will caputure my defined timestamp. as of now splunk is getting the time and date somewhere in my logs which is not my defined pattern or not my desired timestamp to use.