Splunk Search

How to write regex to extract date related subfields from req_time?

dhavamanis
Builder

We have created new sourcetype (acquia_access_combined) by coping the existing sourcetype (access_combined) and added few fields. field values are extracting but the below fields are not showing up, Can you please help me to get these fields out of req_time,

date_hour
date_mday
date_minute
date_month
date_second
date_wday
date_year
date_zone

Log Format configured is :

REGEX = ^[[nspaces:logfilename]]\s++[[nspaces:nodename]]\s++[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[nspaces:bytes]?[[all:other]]

1 Solution

somesoni2
Revered Legend

Based on the sample logs you shared in your prev question.

varnishncsa bal-1234 1.48.1.2 - - [22/Aug/2014:15:04:45 +0000] "GET http://www.test.com/error HTTP/1.1" 404 30041 "http://www.test.com/test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2" 0.090000868 miss pass request_id="v-a732c002-2a0d-11e4-88b7-12313d2d8c3b" "-"

try adding following to your sourcetype definition in props.conf

[acquia_varnish_log]
...other entries...
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z 
TIME_PREFIX = \[

Update:

The entries looks good to me. Can try this workaround (setup calculated fields )

[acquia_varnish_log]
    ...other entries...
EVAL-date_hour = strftime(_time,"%H")
EVAL-date_mday = strftime(_time,"%d")
EVAL-date_minute = strftime(_time,"%M")
EVAL-date_month = strftime(_time,"%H")
EVAL-date_second = strftime(_time,"%S")
EVAL-date_wday = strftime(_time,"%A")
EVAL-date_year = strftime(_time,"%Y")
EVAL-date_zone = strftime(_time,"%z")

View solution in original post

somesoni2
Revered Legend

Based on the sample logs you shared in your prev question.

varnishncsa bal-1234 1.48.1.2 - - [22/Aug/2014:15:04:45 +0000] "GET http://www.test.com/error HTTP/1.1" 404 30041 "http://www.test.com/test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2" 0.090000868 miss pass request_id="v-a732c002-2a0d-11e4-88b7-12313d2d8c3b" "-"

try adding following to your sourcetype definition in props.conf

[acquia_varnish_log]
...other entries...
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z 
TIME_PREFIX = \[

Update:

The entries looks good to me. Can try this workaround (setup calculated fields )

[acquia_varnish_log]
    ...other entries...
EVAL-date_hour = strftime(_time,"%H")
EVAL-date_mday = strftime(_time,"%d")
EVAL-date_minute = strftime(_time,"%M")
EVAL-date_month = strftime(_time,"%H")
EVAL-date_second = strftime(_time,"%S")
EVAL-date_wday = strftime(_time,"%A")
EVAL-date_year = strftime(_time,"%Y")
EVAL-date_zone = strftime(_time,"%z")

somesoni2
Revered Legend

Thanks @kristian for pointing that out. Just updated the same.

0 Karma

kristian_kolb
Ultra Champion

regex is not supported for TIME_FORMAT. Change the '\s' for a ' ', i.e. a blank.

/k

dhavamanis
Builder

Thanks, I have tried adding the TIME_FORMAT, but still its not extracting the required fields. please correct me if anything wrong.

[acquia_varnish_log]
MAX_TIMESTAMP_LOOKAHEAD = 128
REPORT-access = acquia-access-extractions
SHOULD_LINEMERGE = False
TIME_FORMAT = %d/%b/%Y:%H:%M:%S\s%z
TIME_PREFIX = \[
0 Karma
Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...