Getting Data In

WebSphere logs and timezone set as EST for Australian user?

Explorer

I am trying to index a WebSphere WAS log, where the time sone format is like this

[24/11/11 10:49:57:538 EST] 0000004a ServletWrapper I SRVE0242I: [custom-webapp] [/app]

  • The EST used here I think means Australian EST not American EST !
  • Splunk reads the log fine BUT ... it treats the 'raw input' time to US EST
  • e.g the entry above which was recorded at 10.49 am is indexed as 11/25/11 2:49:57.538 AM
  • this suggests that Spunk has interpreted this date as US EST raw
  • and 'wound this date/time' forward to its Oz equivalent namely Nov 25th at 2.49 am

What I actually want Splunk to do is treat this date time as Australian 'as is'
- namely the date in the index = exactly the same as the date recorded

I've tried the following changes to props.conf without success

[host::my_host*]
TZ = Australia/Melbourne

I suspect that Splunk sees EST in the input file and assumes its US EST
and then sees TZ and adds the diff between US EST and Oz to the input values
- thus winding my logs entries fwd

I just want them to be treated 'as is'

Can you help ?

1 Solution

Influencer

I don't think the props.conf gets totally bypassed, this doc, mentions that Splunk does look for the date/time in the event, but using the TIME_FORMAT (if present in props.conf).

Have tried defining the TIME_FORMAT in props.conf? - I was also thinking about limiting the "lookahead" to limit how far Splunk looks for the date/time (i.e. stopping it before it gets to the "EST"), for this you would use something like "MAX_TIMESTAMP_LOOKAHEAD = 22" (which should stop the lookahead before the EST).

An example of this can be found here for the TIME_FORMAT=<strptime>. And here for an example using MAX_TIMESTAMP_LOOKAHEAD.

Also to ask an obvious question are you using the "$SPLUNK_HOME/bin/splunk clean", to remove the old incorrect events?

View solution in original post

Contributor

Still trying to get this to work properly. The problem with TZ_ALIAS is that it doesn't work on the right part of the problem. Sure you can change the TZ but WAS has already inserted the wrong timestamp into the event log (14 hours off) so it doesn't help. What is needed to is to rewrite the timestamp in the event (EST) using the offset to the correct timestamp (system time, in this case AEST) so that they agree.

Anyone got a recipe for this? Don't want to reinvent the wheel if I don't have to. Alternatively I'm thinking I might be able to simply discard the time in the event, or perhaps extract it as a field and then replace it with _indextime to hide the wrong values. Ugly, and should not be necessary.

Or IBM could make system time and not Armonk time the default in WAS. Yeah, and I will keep a lookout for articles about porcine aviation in the newspapers 🙂

cm

0 Karma

Contributor

Yes and no minister. I tried TZ_ALIAS yesterday and nothing changed. Can we please have some more detail on how to deploy this change, for those of us who do not have a WAS installation to mess around with.

Does it need to go in the UFs? In the appliance? On the indexer? Can it be set globally or do we have to edit each and every source or sourcetype that the WAS app uses? And when is the solution going to be built in to the app?? Looks like this problem has been around for quite a while.

Charles

0 Karma

Motivator

As of Splunk 5.0.2 this problem has been resolved!

You can now set what timezone offset an acronym like "EST" refers to - in props.conf

http://docs.splunk.com/Documentation/Splunk/latest/Data/ApplyTimezoneOffsetsToTimeStamps

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

(Search the props.conf manual page for "Australia" and you'll find the right section)

0 Karma

Contributor

This needs to be added to the documentation as a 'gotcha'. WSA uses EST UNITED STATES by default if you don't set a time zone in the WS environment variables. It's NOT EST Australia. Changing this value once WSA apps are deployed may create unknown side effects, according to the websphere guys I'm talking to.

Note that the end result of this for diligent system admins who've set up ntp is that the time zone in the events does not agree with the system time that Splunk ingests the event on. This causes a lot of interesting problems in the app, as I have just discovered at my customer site.

Can we please have an official, tested and supported solution for this problem which does not involve messing around with WSA, as that would typically involve change control, outages, and maybe even app rewrites which might be out of the question.

0 Karma