topic Re: WebSphere logs and timezone set as EST for Australian user? in Getting Data In

WebSphere logs and timezone set as EST for Australian user?

miken_hg5 — Mon, 28 Nov 2011 10:24:31 GMT

I am trying to index a WebSphere WAS log, where the time sone format is like this

[24/11/11 10:49:57:538 EST] 0000004a ServletWrapper I SRVE0242I: [custom-webapp] [/app]

The EST used here I think means Australian EST not American EST !
Splunk reads the log fine BUT ... it treats the 'raw input' time to US EST
e.g the entry above which was recorded at 10.49 am is indexed as 11/25/11 2:49:57.538 AM
this suggests that Spunk has interpreted this date as US EST raw
and 'wound this date/time' forward to its Oz equivalent namely Nov 25th at 2.49 am

What I actually want Splunk to do is treat this date time as Australian 'as is'
- namely the date in the index = exactly the same as the date recorded

I've tried the following changes to props.conf without success

[host::my_host*]
TZ = Australia/Melbourne

I suspect that Splunk sees EST in the input file and assumes its US EST
and then sees TZ and adds the diff between US EST and Oz to the input values
- thus winding my logs entries fwd

I just want them to be treated 'as is'

Can you help ?

Re: WebSphere logs and timezone set as EST for Australian user?

miken_hg5 — Mon, 28 Nov 2011 10:48:55 GMT

PS: One option I have considered is this

that Splunk in order of preference sees the EST entry and assumes it is US/EST, and applies that by default

The documentation says it evaluates timezone in this order

a. seeing values in the log like EST
b. then using values in props
c. then using time set on the server

So I think in my case (a) is being applied before it even considers b, hence my mods to props.cong are redundant

So ... how I teach Splunk to recognise Oz style timezones ??

Re: WebSphere logs and timezone set as EST for Australian user?

MHibbin — Mon, 28 Nov 2011 11:07:44 GMT

I assume you have restarted the services, I also assume you are using the host name used in the events (either configured in inputs.conf or the default in your server.conf file).

Or maybe you can use "DATETIME_CONFIG = CURRENT" in your props.conf (check http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf for more info), this should assign the event time as the system time. Is this any help?

Regards,

Re: WebSphere logs and timezone set as EST for Australian user?

miken_hg5 — Tue, 29 Nov 2011 05:01:08 GMT

I have tried the above without success 😞

By default, Splunk applies time zones using these rules, in this order:

FIRSTLY. Splunk uses any time zone specified in raw event data (for example, PST, -0800).

And my log has an eyecatching EST in it !
e.g. [24/11/11 10:49:57:538 EST] 0000004a ServletWrapper I SRVE0242I: [custom-webapp] [/app]

Then ... Splunk uses the value of a TZ attribute set in props.conf, if the event matches the host, source, or source type specified by the stanza.

And then ... Splunk uses the time zone of the Splunk server that indexes the event.

So I suspect my props.conf is being bypassed because it sees EST in there already.
My host name matches the wildcard in the props stanza

Re: WebSphere logs and timezone set as EST for Australian user?

MHibbin — Tue, 29 Nov 2011 10:47:52 GMT

I don't think the props.conf gets totally bypassed, this doc, mentions that Splunk does look for the date/time in the event, but using the TIME_FORMAT (if present in props.conf).

Have tried defining the TIME_FORMAT in props.conf? - I was also thinking about limiting the "lookahead" to limit how far Splunk looks for the date/time (i.e. stopping it before it gets to the "EST"), for this you would use something like "MAX_TIMESTAMP_LOOKAHEAD = 22" (which should stop the lookahead before the EST).

An example of this can be found here for the TIME_FORMAT=<strptime>. And here for an example using MAX_TIMESTAMP_LOOKAHEAD.

Also to ask an obvious question are you using the "$SPLUNK_HOME/bin/splunk clean", to remove the old incorrect events?

Re: WebSphere logs and timezone set as EST for Australian user?

miken_hg5 — Mon, 28 Sep 2020 10:11:07 GMT

Sorted 😉

The change to using MAX_TIMESTAMP_LOOKAHEAD was the key

without it the system always gave preference to the EST and assumed data was from the US EST zone. Now it stores 'as is' in Oz time

Re: WebSphere logs and timezone set as EST for Australian user?

miken_hg5 — Sat, 03 Dec 2011 07:17:47 GMT

Sorted using the tip on

MAX_TIMESTAMP_LOOKAHEAD = 22

Re: WebSphere logs and timezone set as EST for Australian user?

MHibbin — Sat, 03 Dec 2011 09:31:24 GMT

Awesome, can you make the answer as accepted please? Thanks.

Re: WebSphere logs and timezone set as EST for Australian user?

MHibbin — Sat, 03 Dec 2011 09:31:33 GMT

Awesome, can you make the answer as accepted please? Thanks.

Re: WebSphere logs and timezone set as EST for Australian user?

Parameshwara — Mon, 28 Sep 2020 12:46:49 GMT

Need assistance as I think I'm not doing this right. I'm using Splunk 5 for windows.

Here's what my log file looks like: e.g. ####<*Nov 12, 2012 4:04:35 PM CST>

Right now Splunk catches CST and translates into US time, so it shows as 11/13/12 9:00:14.000 AM.

There was no props.conf in my C:\Program Files\Splunk\etc\system\local directory so created one:

[source::*]
TZ = Australia/Adelaide
TIME_PREFIX = ####<
TIME_FORMAT = %b %d, %Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 28

I am sure it must have been my TIME_PREFIX & TIME_FORMAT formatting error. Can anyone check and confirm my sytax? Cheers!

Re: WebSphere logs and timezone set as EST for Australian user?

Parameshwara — Wed, 14 Nov 2012 03:54:36 GMT

Found 2 other alternatives! Posting as an answer.

Re: WebSphere logs and timezone set as EST for Australian user?

Parameshwara — Wed, 14 Nov 2012 04:11:24 GMT

I tried configuring the props.conf by defining parameter for a particular source, but didn't work. Managed to find 2 other alternatives!

1st solution:
As all my log files are going to be on Australia timezone, simplest solution was to remove CST & EST from datetime.xml. That's it.

2nd solution:
Defined a new source time that ignores CST. Steps to do this:
1.Go to Manager > Data inputs > Files & directories > New
2.Select option 'Preview data before indexing'
3.Select the source type.
4.You would see that the time stamp selected/highlighted would include CST/EST.
5.Select 'adjust timestamp and event break settings'
6.Select 'Timestamps' tab
7.Select option 'Timestamp never extends more than xxx chars into the event', and define the right number so that CST/EST is ignored.
8.After applying your new configuration and verified it ignores CST/EST, select 'Continue' and save it as a new source type.
9.Import logs under new source type and it should work.

Re: WebSphere logs and timezone set as EST for Australian user?

cmeo — Fri, 24 May 2013 05:47:57 GMT

This needs to be added to the documentation as a 'gotcha'. WSA uses EST UNITED STATES by default if you don't set a time zone in the WS environment variables. It's NOT EST Australia. Changing this value once WSA apps are deployed may create unknown side effects, according to the websphere guys I'm talking to.

Note that the end result of this for diligent system admins who've set up ntp is that the time zone in the events does not agree with the system time that Splunk ingests the event on. This causes a lot of interesting problems in the app, as I have just discovered at my customer site.

Can we please have an official, tested and supported solution for this problem which does not involve messing around with WSA, as that would typically involve change control, outages, and maybe even app rewrites which might be out of the question.

Re: WebSphere logs and timezone set as EST for Australian user?

Jason — Wed, 29 May 2013 11:49:44 GMT

As of Splunk 5.0.2 this problem has been resolved!

You can now set what timezone offset an acronym like "EST" refers to - in props.conf

http://docs.splunk.com/Documentation/Splunk/latest/Data/ApplyTimezoneOffsetsToTimeStamps

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

(Search the props.conf manual page for "Australia" and you'll find the right section)

Re: WebSphere logs and timezone set as EST for Australian user?

cmeo — Fri, 19 Jul 2013 00:16:06 GMT

Yes and no minister. I tried TZ_ALIAS yesterday and nothing changed. Can we please have some more detail on how to deploy this change, for those of us who do not have a WAS installation to mess around with.

Does it need to go in the UFs? In the appliance? On the indexer? Can it be set globally or do we have to edit each and every source or sourcetype that the WAS app uses? And when is the solution going to be built in to the app?? Looks like this problem has been around for quite a while.

Charles

Re: WebSphere logs and timezone set as EST for Australian user?

cmeo — Fri, 26 Jul 2013 05:42:29 GMT

Still trying to get this to work properly. The problem with TZ_ALIAS is that it doesn't work on the right part of the problem. Sure you can change the TZ but WAS has already inserted the wrong timestamp into the event log (14 hours off) so it doesn't help. What is needed to is to rewrite the timestamp in the event (EST) using the offset to the correct timestamp (system time, in this case AEST) so that they agree.

Anyone got a recipe for this? Don't want to reinvent the wheel if I don't have to. Alternatively I'm thinking I might be able to simply discard the time in the event, or perhaps extract it as a field and then replace it with _indextime to hide the wrong values. Ugly, and should not be necessary.

Or IBM could make system time and not Armonk time the default in WAS. Yeah, and I will keep a lookout for articles about porcine aviation in the newspapers 🙂