Problem with timestamp in props.conf file.

[dwfarris]

Here is a sample log record. . .
[Fri, 25 May 2018 17:07:34GMT] [some_named_plugin.dll] [Process:4856][ERROR] : invalid token for user ZYX1\john_xyxxy

Now the time above has GMT beside it, but the actual time is US/Easter time. . .

Well, I take one record and "Add Data" with this 1 line text file just to play with the format and get the stanza to put in the props.conf file.

I create a time_format that works here. But when I put that stanza into the props.conf file, the time comes out 4 hours off. . .

For the one record above, the splunk time SHOULD be 17:07:34. Eastern. like in "Add Data" test record. I just want to ignore brackets, and GMT string.

But it comes out as 13:07:34.

Again, time_format works perfectly in "Add Data". but not in actual props.conf.

[. . . something . . .]
ANNOTATE_PUNCT = false
KV_MODE = auto
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
TIME_FORMAT=[%a, %d %b %Y %H:%M:%SGMT]
TIME_PREFIX = ^
TRUNCATE = 999999
pulldown_type = 1

Again, this stanza works fine in "Add Data" single record test. but not with the actual log records. It is 4 hours off.

The server is US/Eastern time. . . Everything on this splunk should be in US/Eastern time. . .

I am testing various time zone offsets to get it correct, but I dont think that is the CORRECT fix.

Why does it work in "Add Data" but not props.conf?

Thanks
David

[sloshburch]

If it works in the UI but not in the configuration then this might be as simple as where the configuration lives.

The time is selected at the first point it's cooked (I think). If the props.conf you created is just in the Search Head (the UI) then it would make sense that this is not working. You need this props.conf to live at the indexer or whereever the data gets cooked first. So, if it's collected with a heavy forwarder, this would need to go there. If it's collected with a Universal Forwarder then make sure these props configs are on the indexers.

If that doesn't solve it, would you let us know your deployment set up so we can take that into account as to why it worked in the UI but not in the conf?

Another option might be to use a SED cmd to replace the GMT with EST (or nothing at all). I forget if that happens before the timestamp is calculated (I think it does).

[dwfarris]

Ok, wanted to update with more info. . . And thanks for suggestions. . .

On first reply from Niketnilay. . .
Actually, here, ALL servers, ALL users default time zone is US/Eastern. . . regardless of where they are, we keep everything set to US/Eastern.

The server producing these records is set to US/Eastern. So the time is really US/Eastern.

The problem is that an application (that we do not own the code for) is dumping its logs and is adding the GMT. even though the time is eastern. We cant seem to get rid of the GMT so I want the time_format to ignore the GMT and just read the time as eastern.

Your prefix/format was one I had tried as well. . . I have tried several. . .

See next update for more current status.

[somesoni2]

If your logged time is indeed EST (or simply NOT GMT), try this a your props.conf configuration

[. . . something . . .]
LINE_BREAKER = ([\r\n]+)(?=\[\w+,\s\d+)
SHOULD_LINEMERGE=true
TIME_FORMAT=%d %b %Y %H:%M:%S
TIME_PREFIX = ^\[\w+,\s
MAX_TIMESTAMP_LOOKAHEAD = 20
TRUNCATE = 999999

[dwfarris]

Next, Thanks Somesoni2,
1. Had not tried to skip past the Fri, with the line break. Unfortunately that did not work either.

1. What I did realize after trying these as well is that the problem is simpler.

2. It appears that NOTHING I try for time_format actually does anything. I can just swap the %d, %Y, %H, %M around in any order and the indexed record never changes regardless of what format I have.

3. Since the time is posting 4 hours off of US/Eastern, I also started trying several TZ= timezones. . . US/Eastern, US/Central, . . . Alaska, Virgin Islands, Germany. . . JUST to try to get the indexed time to change.

Regardless of time_format. . . Regardless of TZ=. . . The posted index time never changes. . . Always 4 hours off.
1. I then decided it must be using some default somewhere. . . So I changed these to go to a new/different sourcetype. . . Same issue.

2. This is strange, because in this SAME props.conf file, there is another sourcetype that I had to change the TZ to UTC to correct a DIFFERENT issue. This worked as it should have. . . Just not for this GTM stanza. . .

Any suggestions
Thanks

[niketn]

@dwfarris, the timezone on your raw event seems to be GMT. Could that be possible reason for 4 hour's difference?

Have you checked Logged in User's Time Zone? Set to EST to have the time i.e. _time reflected in EST instead of GMT.

Also I felt slight change for TIME_PREFIX and TIME_FORMAT, please also test these in preview mode or with sample data. %Z is for Time Zone which is GMT as per your data sample.

TIME_FORMAT=%a, %d %b %Y %H:%M:%S%Z
TIME_PREFIX=^\[
MAX_TIMESTAMP_LOOKAHEAD=30

[dwfarris]

Thanks. . .

Here, ALL SERVERS, ALL USERS, ALL APPLICATIONS are set to US/Eastern time regardless of location/time zone.

So this server is set to US/Eastern. Problem is the application that is writing this log record want to spit out GMT along with the time in eastern. . .

So I am just trying to ignore the GMT.

I had tried this prefix/format as well with no luck. . . see next answer comments below for more info.

Thanks