log4j format log files and timezone setting not working

[BradTaylor]

revised as requested for better background information

Hi I have a newb time zone question.

What have I configured incorrectly that is preventing splunk from applying the TZ rules defined in props.conf to index UTC time zone files correctly?

I've set up a props.conf file with a rule that defines the servers to default to Canada/Mountain and then specifies UTC time zone for log4j files.

I was going to add a [sourcetype::log4j_appian] stanza to the props.conf but I believe according to the precedence rules described in the manual that the [host::abserver*] stanza will override that value anyway, so I was forced to use the source keyword stanza.

http://www.splunk.com/base/Documentation/latest/Admin/Applytimezoneoffsetstotimestamps http://www.splunk.com/base/Documentation/latest/admin/Propsconf

Precedence:

For settings that are specified in multiple categories of matching stanzas, [host::] spec settings override [] spec settings. Additionally, [source::] and [] settings.

 [t807309@abserver-web local]$ cat props.conf
 [rule::access_common_vhost]
 sourcetype = access_common_vhost
 #access_common_vhost:   some.virtual.host 204.191.153.144 - -[05/May/2010:21:50:01 -0700] "GET /arsys/shared/images/login_image.jpg HTTP/1.1" 200 21617
 #access_common:         204.191.153.144 - - [05/May/2010:21:50:01 -0700] "GET /arsys/shared/images/login_image.jpg HTTP/1.1" 200 21617
 #MORE_THAN_75 = ^\S+ \S+ \S+ \[[^\]]+\] "[^"]+" \S+ \S+$
 MORE_THAN_75 = ^\S+ \S+ \S+ \S+ \[[^\]]+\] "[^"]+" \S+ \S+$

 [host::abserver*]
 TZ = Canada/Mountain

 [source::/appian/logs/*.log]
 TZ = UTC

server #1: abserver-eng:

• server in Canada/Mountain timezone
• has props.conf
• index server

server #2: abserver-app:

• server in Canada/Mountain timezone
• has props.conf
• standard forwarder; will become light forwarder
• weblogic server (weblogic_stdout)
• log4j log files with custom sourcetype (log4j_appian) assigned

[t807309@abserver-app local]$ cat inputs.conf
[monitor:///opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log]
disabled = false
followTail = 0
index = main
sourcetype = weblogic_stdout

[monitor:///opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_error.log]
disabled = false
followTail = 0
index = main
sourcetype = weblogic_stderr

[monitor:///appian/logs/*.log]
disabled = false
followTail = 0
index = main
sourcetype = log4j_appian

Samples:

/opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log

• mixed mode;

• Weblogic lines have GMT stamp:

• log4j format; no TZ stamp; GMT:

  2010-04-23 21:08:07,434 [Main Thread] DEBUG com.appiancorp.kougar.mapper.parameters.ArrayParameterConverter - performing item-by-item conversion of return value <[Lcom.appiancorp.suiteapi.process.TypedVariable;@2575e61> to

/opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_error.log

• no TZ stamp; looks like GMT

  May 5, 2010 9:13:22 PM com.metaparadigm.jsonrpc.JSONRPCBridge registerLocalArgResolver INFO: registered local arg resolver com.metaparadigm.jsonrpc.JSONRPCBridgeServletArgResolver for local class com.metaparadigm.jsonrpc.JSONRPCBridge with context javax.servlet.http.HttpServletRequest javax.servlet.ServletException: Could not find the config file: /WEB-INF/decorators.xml

/appian/logs/application-server.log

• log4j format; no TZ stamp; GMT:

  2010-05-08 01:16:46,393 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.asi.components.grid.internal.GridAction - The Forum you are attempting to interact with has either been deleted or does not exist. com.appiancorp.asi.components.common.WebComponentException: The Forum you are attempting to interact with has either been deleted or does not exist.

server #3: abserver-web:

• server in Canada/Mountain timezone
• has props.conf
• standard forwarder; will become light forwarder
• apache web server (log file is a variation on access_common, with virtual host name prepended to each line and apache_error logs)

[t807309@abserver-web local]$ cat inputs.conf
[monitor:///var/log/httpd/*_error_log_current]
disabled = false  
followTail = 0 
index = main 
sourcetype = apache_error

[monitor:///var/log/httpd/*_access_log_current]
disabled = false 
followTail = 0 
index = main
sourcetype = access_common_vhost

Samples:

/var/log/httpd/vhost_F5_80_error_log_current

• no TZ stamp

  [Wed May 05 16:03:58 2010] [error] FAILOVER_REQUIRED [line 483 of ap_proxy.cpp]: Service Unavailable

/var/log/httpd/vhost_F5_80_access_log_current

• standard apache time format

  [t807309@abserver-web 05-May]$ tail vhost_F5_80_access_log_2010-05-10 abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:00 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.11 - - [10/May/2010:12:22:03 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:05 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.11 - - [10/May/2010:12:22:08 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:10 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956

The server TZ is set correctly:

[t807309@abserver-app splunk]$ date
Fri May  7 17:57:42 MDT 2010

Here are two sample lines from each of the log files:

[t807309@abserver-app splunk]$ tail /appian/logs/application-server.log
2010-05-07 23:27:21,245 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.ap2.PortalResponse - Error: 404


[t807309@abserver-app splunk]$ tail /opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log
2010-05-07 23:27:21,245 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.ap2.PortalResponse - Error: 404

Both of these events are stamped as 11:27:21 pm, date_zone=-360 (MDT)

Here's what I see in splunk:

http://www.freeimagehosting.net/uploads/1583444cc8.gif

The only thing I am doing outside the box is assigning a different sourcetype (log4j_appian) to the /appian/logs/*.log files. When I look at the events, Splunk has correctly parsed the timestamps however, so I assume no further definition is required.

Here's the inputs.conf stanza that defines the appian log files:

[monitor:///appian/logs/*.log]
disabled = false
followTail = 0
index = main
sourcetype = log4j_appian

Do I need to do more in terms of defining the custom sourcetype for Splunk to be able to assign the correct TZ?

What (else) am I doing wrong here?

thanks...

[gregbujak]

Since you have been provided the Splunk centric advice/approach, let me throw this in the mix...

Have you considered enhancing your conversion pattern to include the TZ? Whenever I have global (or regional) application servers, I go with the enriching the conversionpattern to include the TZ information. It ends up being a lot less of a headache.

[wrangler2x]

The key is to remember when doing transforms is to do your props/transforms where parsing occurs.
After parsing occurs, the data is "cooked" (in Splunk lingo) and anything further you've specified just won't occur. Heavy (or "Universal") Forwarders do parsing, and if this is what is forwarding log entries to you then when the data arrives on the indexer, it is already cooked. You cannot transform cooked data.

So if you add a props.conf file on the forwarder using the sourcetype [log4j_appian] then add a TZ = UTC that should solve the problem.

[BradTaylor]

From my read of the precendence rules, I had to add the [source] stanza, since I have a [host] stanza that defaults logs from the servers to be Canada/Mountain timezone.

[gkanapathy]

I do think that the only TZ config you should have though, should just be for sourcetype [log4j_appian] where you set TZ = UTC. That should go in props.conf whereever the parsing queue is (not on a light forwarder, yes on a heavy forwarder, won't hurt to put it everywhere)

[gkanapathy]

It is hard for me to follow what you have written. Can you explicitly tell us the time zone of each log file and host explictly? Is it basically the case that all files are logging in UTC? Or are there some that are logged in Mountain? It would also help you explictly said which ones Splunk is getting wrong.