All Apps and Add-ons

Timestamps Ignoring Timezone

bengoerz
Explorer

I am having a problem where _time is extracted in the wrong timezone.

My McAfee Web Gateway is in CDT (TZ = Americas/Chicago), but _time is being extracted in UTC.

Here's an example event where Splunk shows _time = 12/8/17 2:18:58.000 AM:

Dec  8 08:18:58 usproxy43 mwg: McAfeeWG|time_stamp=[08/Dec/2017:08:18:58 -0600]|auth_user=User123|src_ip=172.16.0.2|server_ip=123.234.123.234|host=google.com|url_port=443|status_code=200|bytes_from_client=9247|bytes_to_client=415|categories=Search Engines|rep_level=Minimal Risk|method=POST|url=https://google.com/|media_type=application/x-empty|application_name=|user_agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36|block_res=0|block_reason=|virus_name=|hash=|filename=upload|filesize=0|

I was unsuccessful at trying to fix using TZ in props.conf on the Universal Forwarder:

[host::*proxy*]
TZ = America/Chicago

Is my problem with the TZ on the forwarder, or something in the app?

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Add the following to your props.conf file on the indexers, not the universal forwarders:

[host::*proxy*]
TIME_FORMAT=%d/%b/%Y:%T %z
TIME_PREFIX=\[

It has probably been taking the first date, instead of the one with later in the event that has the timezone information.

bengoerz
Explorer

Thanks. I'm on Cloud, so going to take a while to get indexer changes implemented.

Out of curiosity, why don't we just do traditional field extraction on the Search Head?

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

What do you consider traditional field extraction? Splunk best practice is to do field extraction on the search heads.

0 Karma

bengoerz
Explorer

Right. This app does a bunch of fancy transforms, instead of just doing field extraction on the search heads per Splunk best practice. Why?

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

What is the app? Is it from Splunkbase? Sometimes a developer wants to have all the fields extracted to make their searches faster (like if you need tstats speed), which is fine, UNTIL something changes in the format or similar change. It can also be a load on the indexers, which isn't good.

0 Karma

bengoerz
Explorer

The app is Splunk Add-on for McAfee Web Gateway (http://apps.splunk.com/app/3009/).

BTW, I just created my own field extraction on the search heads as an alternate to this app. Posting the regex for posterity or comments:

time_stamp=\[(?P<time>[^"]*)]\|auth_user=(?P<user>[^"]*)\|src_ip=(?P<src_ip>[^"]*)\|server_ip=(?P<dest_ip>[^"]*)\|host=(?P<dest>[^"]*)\|url_port=(?P<dest_port>[^"]*)\|status_code=(?P<status>[^"]*)\|bytes_from_client=(?P<bytes_out>[^"]*)\|bytes_to_client=(?P<bytes_in>[^"]*)\|categories=(?P<category>[^"]*)\|rep_level=(?P<severity>[^"]*)\|method=(?P<http_method>[^"]*)\|url=(?P<url>[^"]*)\|media_type=(?P<http_content_type>[^"]*)\|application_name=(?P<application_name>[^"]*)\|user_agent=(?P<http_user_agent>[^"]*)\|block_res=(?P<action>[^"]*)\|block_reason=(?P<block_reason>[^"]*)\|virus_name=(?P<signature>[^"]*)\|hash=(?P<file_hash>[^"]*)\|filename=(?P<file_name>[^"]*)\|filesize=(?P<filesize>[^"]*)\|
0 Karma

saurabh_tek11
Communicator

@cpetterborg - i tried to do the solution in my test instance.
i am in Dubai which is UTC+4

My space instance is showing the _time for this event after applying above answered props is

12/8/17
6:18:58.000 PM

Is this correct for my splunk instance to show ?

Considering event has UTC -6 and +4 for my location = event time -2 in my location.
Just want to validate if i am thinking it right way ??

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

That looks correct to me. I started by figuring out what it should be, then comparing with your answer so that I wouldn't be swayed, and I got the same answer. I believe that has worked for you.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!