How to edit props.conf to ignore timezone information?

[jwhughes58]

I've got data with a timestamp that looks like this

[2017-07-06T16:32:38.977-07:00]

In props.conf I have this

TIME_PREFIX = ^\[
MAX_TIMESTAMP_LOOKAHEAD = 24
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 999999
NO_BINARY_CHECK = true
TZ = UTC

What I'm finding is even though I ignore the TZ information in the TIME_FORMAT, the offset is getting added to _time. Unfortunately the data source uses the TZ field to signify where the device is physically located, but the time is actually UTC.

Time:7/7/17 6:34:06.657 PM  
Raw Event:[2017-07-07T11:34:06.657-07:00] [OUD] [TRACE] [OUD-24641552] [PROTOCOL] [host: myhostname.org] [nwaddr: aaa.bbb.ccc.ddd] [tid: 115] [userId: uid] [ecid: c6e497c51cca7a85:-7e0b16c4:15cf320db73:-8000-000000000330e42a,0:22] [result: 0] [category: RES] [conn: 305] [op: 127674] [etime: 2] [nentries: 1] [msgID: 127675] SEARCH

Any ideas on how I can get around this? All the other timestamps worked using Time_FORMAT, but they also have a space between the time and the TZ.

TIA,
Joe

[paulstout · Accepted solution]

Joe,

Because we know this timezone offset is incorrect in the source data as it is annotating the time in which the device is deployed rather than the timezone in which the event occurred, I used SED to strip it out in props.conf and kept the rest of your configuration. It worked on my local Splunk with your line of test data.

I added this just before TIME_PREFIX:

SEDCMD-notz = s/\-[0-9][0-9]:[0-9][0-9]\]/]/

[jwhughes58]

Paul,

Thanks this was almost the solution. I found a typo in my props.conf that was throwing things off. I also found this

SEDCMD-movetz = s/(^.{24})/\1 /

works better. Instead of removing the TZ information it adds a space between the time information and the TZ. Then my TIME_FORMAT worked correctly.

[somesoni2]

What is the timezone of the user doing the search? I see the offset between event time (without offset in raw event) and _time value is 5 hrs and this is correct if user's timezone is in CST/CDT. The _time value adjusts based on who's viewing the data.

[jwhughes58]

I always set my TZ to UTC. The offset between event time and _time value is 7 hours. 11:34:06 plus 7 hours is 18:34:06 or 6:34:06 PM. All my other data feeds work, showing event time and _time value as the same. Just this one with no space seems to grab the offset.