revised as requested for better background information
Hi I have a newb time zone question.
What have I configured incorrectly that is preventing splunk from applying the TZ rules defined in props.conf to index UTC time zone files correctly?
I've set up a props.conf file with a rule that defines the servers to default to Canada/Mountain and then specifies UTC time zone for log4j files.
I was going to add a [sourcetype::log4j_appian] stanza to the props.conf but I believe according to the precedence rules described in the manual that the [host::abserver*] stanza will override that value anyway, so I was forced to use the source keyword stanza.
http://www.splunk.com/base/Documentation/latest/Admin/Applytimezoneoffsetstotimestamps http://www.splunk.com/base/Documentation/latest/admin/Propsconf
Precedence:
For settings that are specified in multiple categories of matching stanzas, [host::] spec settings override [] spec settings. Additionally, [source::] and [] settings.
[t807309@abserver-web local]$ cat props.conf
[rule::access_common_vhost]
sourcetype = access_common_vhost
#access_common_vhost: some.virtual.host 204.191.153.144 - -[05/May/2010:21:50:01 -0700] "GET /arsys/shared/images/login_image.jpg HTTP/1.1" 200 21617
#access_common: 204.191.153.144 - - [05/May/2010:21:50:01 -0700] "GET /arsys/shared/images/login_image.jpg HTTP/1.1" 200 21617
#MORE_THAN_75 = ^\S+ \S+ \S+ \[[^\]]+\] "[^"]+" \S+ \S+$
MORE_THAN_75 = ^\S+ \S+ \S+ \S+ \[[^\]]+\] "[^"]+" \S+ \S+$
[host::abserver*]
TZ = Canada/Mountain
[source::/appian/logs/*.log]
TZ = UTC
server #1: abserver-eng:
server #2: abserver-app:
[t807309@abserver-app local]$ cat inputs.conf [monitor:///opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log] disabled = false followTail = 0 index = main sourcetype = weblogic_stdout [monitor:///opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_error.log] disabled = false followTail = 0 index = main sourcetype = weblogic_stderr [monitor:///appian/logs/*.log] disabled = false followTail = 0 index = main sourcetype = log4j_appian
Samples:
/opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log
Weblogic lines have GMT stamp:
log4j format; no TZ stamp; GMT:
2010-04-23 21:08:07,434 [Main Thread] DEBUG com.appiancorp.kougar.mapper.parameters.ArrayParameterConverter - performing item-by-item conversion of return value <[Lcom.appiancorp.suiteapi.process.TypedVariable;@2575e61> to
/opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_error.log
no TZ stamp; looks like GMT
May 5, 2010 9:13:22 PM com.metaparadigm.jsonrpc.JSONRPCBridge registerLocalArgResolver INFO: registered local arg resolver com.metaparadigm.jsonrpc.JSONRPCBridgeServletArgResolver for local class com.metaparadigm.jsonrpc.JSONRPCBridge with context javax.servlet.http.HttpServletRequest javax.servlet.ServletException: Could not find the config file: /WEB-INF/decorators.xml
/appian/logs/application-server.log
log4j format; no TZ stamp; GMT:
2010-05-08 01:16:46,393 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.asi.components.grid.internal.GridAction - The Forum you are attempting to interact with has either been deleted or does not exist. com.appiancorp.asi.components.common.WebComponentException: The Forum you are attempting to interact with has either been deleted or does not exist.
server #3: abserver-web:
[t807309@abserver-web local]$ cat inputs.conf [monitor:///var/log/httpd/*_error_log_current] disabled = false followTail = 0 index = main sourcetype = apache_error [monitor:///var/log/httpd/*_access_log_current] disabled = false followTail = 0 index = main sourcetype = access_common_vhost
Samples:
/var/log/httpd/vhost_F5_80_error_log_current
no TZ stamp
[Wed May 05 16:03:58 2010] [error] FAILOVER_REQUIRED [line 483 of ap_proxy.cpp]: Service Unavailable
/var/log/httpd/vhost_F5_80_access_log_current
standard apache time format
[t807309@abserver-web 05-May]$ tail vhost_F5_80_access_log_2010-05-10 abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:00 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.11 - - [10/May/2010:12:22:03 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:05 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.11 - - [10/May/2010:12:22:08 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956 abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:10 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956
The server TZ is set correctly:
[t807309@abserver-app splunk]$ date
Fri May 7 17:57:42 MDT 2010
Here are two sample lines from each of the log files:
[t807309@abserver-app splunk]$ tail /appian/logs/application-server.log
2010-05-07 23:27:21,245 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.ap2.PortalResponse - Error: 404
[t807309@abserver-app splunk]$ tail /opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log
2010-05-07 23:27:21,245 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.ap2.PortalResponse - Error: 404
Both of these events are stamped as 11:27:21 pm, date_zone=-360 (MDT)
Here's what I see in splunk:
http://www.freeimagehosting.net/uploads/1583444cc8.gif
The only thing I am doing outside the box is assigning a different sourcetype (log4j_appian) to the /appian/logs/*.log files. When I look at the events, Splunk has correctly parsed the timestamps however, so I assume no further definition is required.
Here's the inputs.conf stanza that defines the appian log files:
[monitor:///appian/logs/*.log]
disabled = false
followTail = 0
index = main
sourcetype = log4j_appian
Do I need to do more in terms of defining the custom sourcetype for Splunk to be able to assign the correct TZ?
What (else) am I doing wrong here?
thanks...
Since you have been provided the Splunk centric advice/approach, let me throw this in the mix...
Have you considered enhancing your conversion pattern to include the TZ? Whenever I have global (or regional) application servers, I go with the enriching the conversionpattern to include the TZ information. It ends up being a lot less of a headache.
The key is to remember when doing transforms is to do your props/transforms where parsing occurs.
After parsing occurs, the data is "cooked" (in Splunk lingo) and anything further you've specified just won't occur. Heavy (or "Universal") Forwarders do parsing, and if this is what is forwarding log entries to you then when the data arrives on the indexer, it is already cooked. You cannot transform cooked data.
So if you add a props.conf file on the forwarder using the sourcetype [log4j_appian] then add a TZ = UTC that should solve the problem.
From my read of the precendence rules, I had to add the [source] stanza, since I have a [host] stanza that defaults logs from the servers to be Canada/Mountain timezone.
I do think that the only TZ config you should have though, should just be for sourcetype [log4j_appian]
where you set TZ = UTC
. That should go in props.conf whereever the parsing queue is (not on a light forwarder, yes on a heavy forwarder, won't hurt to put it everywhere)
It is hard for me to follow what you have written. Can you explicitly tell us the time zone of each log file and host explictly? Is it basically the case that all files are logging in UTC? Or are there some that are logged in Mountain? It would also help you explictly said which ones Splunk is getting wrong.