Splunk Search

How to write regex to extract date related subfields from req_time?

dhavamanis
Builder

We have created new sourcetype (acquia_access_combined) by coping the existing sourcetype (access_combined) and added few fields. field values are extracting but the below fields are not showing up, Can you please help me to get these fields out of req_time,

date_hour
date_mday
date_minute
date_month
date_second
date_wday
date_year
date_zone

Log Format configured is :

REGEX = ^[[nspaces:logfilename]]\s++[[nspaces:nodename]]\s++[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[nspaces:bytes]?[[all:other]]

1 Solution

somesoni2
Revered Legend

Based on the sample logs you shared in your prev question.

varnishncsa bal-1234 1.48.1.2 - - [22/Aug/2014:15:04:45 +0000] "GET http://www.test.com/error HTTP/1.1" 404 30041 "http://www.test.com/test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2" 0.090000868 miss pass request_id="v-a732c002-2a0d-11e4-88b7-12313d2d8c3b" "-"

try adding following to your sourcetype definition in props.conf

[acquia_varnish_log]
...other entries...
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z 
TIME_PREFIX = \[

Update:

The entries looks good to me. Can try this workaround (setup calculated fields )

[acquia_varnish_log]
    ...other entries...
EVAL-date_hour = strftime(_time,"%H")
EVAL-date_mday = strftime(_time,"%d")
EVAL-date_minute = strftime(_time,"%M")
EVAL-date_month = strftime(_time,"%H")
EVAL-date_second = strftime(_time,"%S")
EVAL-date_wday = strftime(_time,"%A")
EVAL-date_year = strftime(_time,"%Y")
EVAL-date_zone = strftime(_time,"%z")

View solution in original post

somesoni2
Revered Legend

Based on the sample logs you shared in your prev question.

varnishncsa bal-1234 1.48.1.2 - - [22/Aug/2014:15:04:45 +0000] "GET http://www.test.com/error HTTP/1.1" 404 30041 "http://www.test.com/test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2" 0.090000868 miss pass request_id="v-a732c002-2a0d-11e4-88b7-12313d2d8c3b" "-"

try adding following to your sourcetype definition in props.conf

[acquia_varnish_log]
...other entries...
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z 
TIME_PREFIX = \[

Update:

The entries looks good to me. Can try this workaround (setup calculated fields )

[acquia_varnish_log]
    ...other entries...
EVAL-date_hour = strftime(_time,"%H")
EVAL-date_mday = strftime(_time,"%d")
EVAL-date_minute = strftime(_time,"%M")
EVAL-date_month = strftime(_time,"%H")
EVAL-date_second = strftime(_time,"%S")
EVAL-date_wday = strftime(_time,"%A")
EVAL-date_year = strftime(_time,"%Y")
EVAL-date_zone = strftime(_time,"%z")

somesoni2
Revered Legend

Thanks @kristian for pointing that out. Just updated the same.

0 Karma

kristian_kolb
Ultra Champion

regex is not supported for TIME_FORMAT. Change the '\s' for a ' ', i.e. a blank.

/k

dhavamanis
Builder

Thanks, I have tried adding the TIME_FORMAT, but still its not extracting the required fields. please correct me if anything wrong.

[acquia_varnish_log]
MAX_TIMESTAMP_LOOKAHEAD = 128
REPORT-access = acquia-access-extractions
SHOULD_LINEMERGE = False
TIME_FORMAT = %d/%b/%Y:%H:%M:%S\s%z
TIME_PREFIX = \[
0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...