Hi,
I have events that look like this
192.168.10.124 - - [02/Nov/2016:08:59:59 +0900] "GET /ICHealthCheck/serverstatus HTTP/1.1" 200 2 "-" "a10hm/1.0"
And I need splunk to use 02/Nov/2016:08:59:59 +0900 as the _time value when splunk imports the data. I testing this manually by importing a file and then using the timestamp dropdown in the Set Source Type section.
I have a prefix of \[
in the timestamp prefix field
%d/%b/%Y:%H:%M:%S %z
in the timestamp format field
and Asia/Tokyo +9
in the timezone field
Yet when the data gets imported I get a _time value of 11/1/16 12:00:00.000 AM
for the above event.
There are lots more events that have the same problem.
What am I missing ???
If your events already contain timezone information, I believe you don't have to specify timezone explicitly (TZ attribute in props.conf).
For above data, your sourcetype definition should look like this
[yoursourcetype]
#Timerange related
TIME_PREFIX = ^(\S+\s+){3}\[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = 25
#Linebreaking if you want to use this, or else keep yours
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s)
If your events already contain timezone information, I believe you don't have to specify timezone explicitly (TZ attribute in props.conf).
For above data, your sourcetype definition should look like this
[yoursourcetype]
#Timerange related
TIME_PREFIX = ^(\S+\s+){3}\[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = 25
#Linebreaking if you want to use this, or else keep yours
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s)
Hi Somesoni2!
I think you are very very close! Once I applied your suggestions, I had to made a few modifications because I didn't give you all the info
In addition to the above event there are also events that look like this in the file:
219.110.1.249, 192.168.10.102 - - [01/Nov/2016:18:08:09 +0900] "GET /mp/styles/ext_button.css HTTP/1.1" 200 19010 "https://portal.intelligent-home.jp:10443/mp/customersearch.action" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko"
the TIME_PREFIX and TIME_FORMAT combination didn't work with both settings so I removed the %z at the end. That seemed to get very close (right date but the time was 5 hrs off)
Any other thoughts?
I tried using \[
for the time prefix but then the date and time got all screwed up again.
The time format of the above event still same as one in your question (hope that's not a copy paste error here), you'd need to update time prefix and line breaker. Give this a try
[yoursourcetype]
#Timerange related
TIME_PREFIX = ^(\S+\s+){3,4}\[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = 25
#Linebreaking if you want to use this, or else keep yours
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
Time format was still still the same but there was another IP address in there which threw off the TIME PREFIX regex but I;ll try your new one.... stay tuned 🙂
for whatever reason the %z at the end of TIME_FORMAT is only capturing +090 instead of +0900 and it is giving the parser and me....fits 🙂
using everything on your updated one except for the %z at the end
Log entry 192.168.10.124 - - [02/Nov/2016:08:59:59 +0900]
Splunk _time 11/2/16 3:59:59.000 AM
soooooo close!
My bad, change MAX_TIMESTAMP_LOOKAHEAD to 26. (sorry I replied to you before my coffee, I can see better now)
well that allowed the %z to work.... and yes there is a but coming......
but...... sadly it seems to have regressed back to the event date being the 2nd but splunk time is the 1st
Here is what I have
Timezone=Auto
Timestamp format=%d/%b/%Y:%H:%M:%S %z
Timestamp prefix=^(\S+\s+){3,4}\[
Lookahead=26
LINE_BREAK=([\r\n]+)(?=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s)
And what else I have is a bit less hair now........
my updated LINE_BREAKER doesn't have a \s
at the end, so you may want to fix that as well.
For the 5 Hr difference, you'd see the difference between _time and timestamp on your data which will be equal to the difference between time zone on the data and your user timezone. What is the timezone your user-profile is using?
My user is set to the central timezone (-0600). The data is in JST (+0900).
Ah, missed the \s, I'll get that corrected, thanks!
Well I'll be...... it works! 🙂
TY!!!!
Tokyo (Japan) Wednesday, November 2, 2016 at 8:55:00 AM JST UTC+9 hours
Bloomington (USA - Illinois) Tuesday, November 1, 2016 at 6:55:00 PM CDT UTC-5 hours
Corresponding UTC (GMT) Tuesday, November 1, 2016 at 23:55:00
What is the time that you see for an event with timestamp of 02/Nov/2016:08*
?
oh and the linemerge was already = false
Any other thoughts on how to close the 5hr time difference?
Correct. Timestamps with integrated timezone information are automatically normalized to UTC time on the indexer. Its for showing consistent time data through the environment. In example you have users in different timezones. So the users get set the timestamp to their configured timezone.
Just did a quick and dirty adjustment on my local instance and timestamp started reflecting correctly however we have to understand that a time has three aspects to it:
1) TimeZone in raw logs (Tokyo)
2) TimeZone of Indexer (??)
3) Timezone of User accessing the data (??)
I restarted my local laptop Spluk instance with updating my laptop timezone to Tokyo timezone and also changed my Splunk user timezone to Tokyo in the Splunk > Usernmae> EditAccount > TimeZone. After that when I upload your sample data it detected the way you want it.
Having said that:
This maybe what you want to achieve but is not advisable as Splunk will try to index the data by offsetting timezones depending on which timezones its indexers/users are in so that there are no mismatches and everyone gets a "localized" view.
Hope that it helps.
Can you provide the sourcetype in props.conf to us pls. Its easier to fix problems by having the full information.
Some infos first. Splunk Indexers normalize every Date to UTC time. Just take a look at this answers post.
https://answers.splunk.com/answers/135193/splunk-indexing-and-time-zone-normalization.html
Give me some feedback if this is helpful.
kind regards