topic Re: Why is the log time different than the system time? in Getting Data In

Why is the log time different than the system time?

pdgill314 — Tue, 22 May 2018 17:32:03 GMT

I have been beating my head against the wall on this one for a few days now. I have tried every suggestion I can find...

I have logs that are being generated in Eastern Time on a server. That server's date config is UTC. My Splunk indexers are in UTC. My timezone for my user is in Eastern Time, yet, the logs always show up 4 hours behind.

Example log:

2018-05-22T13:01:06.882,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" []controller.CourseSectionManagementController.processCourseSection() line 70: CourseSectionManagementService Full Request org.apache.catalina.connector.RequestFacade@2ad67b7a 
2018-05-22T13:01:06.973,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" [] service.impl.CourseSectionManagementServiceImpl.processDeleteCourseSectionRequest() line 89:  Message to be sent back to ABCD <200,<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

What Splunk sees:

_time 2018-05-22T09:01:06.882-04:00

5/22/18
9:01:06.882 AM  
2018-05-22T13:01:06.882,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" [] controller.CourseSectionManagementController.processCourseSection() line 70: CourseSectionManagementService Full Request org.apache.catalina.connector.RequestFacade@2ad67b7a

_time 2018-05-22T09:01:06.973-04:00

5/22/18
9:01:06.973 AM  
2018-05-22T13:01:06.973,GMT-04:00 DEBUG "ajp-bio-127.0.0.1-8009-exec-405" [] service.impl.CourseSectionManagementServiceImpl.processDeleteCourseSectionRequest() line 89:  Message to be sent back to ABCD <200,<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

Props.conf file:

[Console]
KV_MODE = xml
DATETIME_CONFIG = NONE
TZ = America/New_York
TZ_ALIAS = GMT=UTC
TIME_FORMAT =  %Y-%m-%dT%H:%M:%S.%Q,%Z%:z
MAX_TIMESTAMP_LOOKAHEAD = 36
BREAK_ONLY_BEFORE_DATE = true

inputs.conf:

[monitor:///apps/tomcat-instances/*/logs/console/Console.log]
ignoreOlderThan = 7d
sourcetype = Console
index = sis
disabled = 0

Any suggestions on what I might be missing?

Thank you.

Re: Why is the log time different than the system time?

somesoni2 — Tue, 22 May 2018 18:03:09 GMT

I would make following changes to props.conf (which goes on Indexer/Heavy forwarder whichever comes first in the data flow),

[Console]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N,GMT%:z
MAX_TIMESTAMP_LOOKAHEAD = 34

Your raw data has time zone in them, so no need to explicitly set it. Your data is not XML so, removing KV_MODE attribute there. (plus it should be set on search heads).

Re: Why is the log time different than the system time?

pdgill314 — Tue, 22 May 2018 18:13:51 GMT

Thank you for replying. Giving that a shot now.

I tried adding what I had for the props.conf onto the indexers, and it worked to correct the time, but merged three events into one, so hopefully your suggestion should help.

There is XML data there but it is in a SOAP envelope, e.g.:

   ...
            <imsx_codeMajor>success</imsx_codeMajor>
            <imsx_severity>warning</imsx_severity>
            <imsx_description>success</imsx_description>
        </imsx_statusInfo>
    </ns2:local>
    ,{}>

Thanks.

Re: Why is the log time different than the system time?

somesoni2 — Tue, 22 May 2018 18:20:11 GMT

Since your data is not pure XML (has some non-xml strings as well), the KV_MODE=xml would not work (which should be set on Search Heads).

Re: Why is the log time different than the system time?

pdgill314 — Tue, 22 May 2018 18:24:39 GMT

Ah. Ok. Thanks for that.

Should I even have a Props.conf for this app on the Universal Forwarder (server generating logs)?

Re: Why is the log time different than the system time?

xpac — Tue, 22 May 2018 19:18:58 GMT

Just to add, if Splunk recognizes a timezone information in the timestamp, any TZ = setting will be ignored/overruled.

Re: Why is the log time different than the system time?

pdgill314 — Tue, 22 May 2018 20:00:06 GMT

Ended up using this in the props.conf on my indexers:

[Console]
SHOULD_LINEMERGE = false
TZ = America/New_York
TZ_ALIAS = GMT=UTC
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%Q,%Z%:z
MAX_TIMESTAMP_LOOKAHEAD = 34

Looks like it is working correctly now.

Thanks for your help!