Splunk Search

How to write regex to extract date related subfields from req_time?

dhavamanis
Builder

We have created new sourcetype (acquia_access_combined) by coping the existing sourcetype (access_combined) and added few fields. field values are extracting but the below fields are not showing up, Can you please help me to get these fields out of req_time,

date_hour
date_mday
date_minute
date_month
date_second
date_wday
date_year
date_zone

Log Format configured is :

REGEX = ^[[nspaces:logfilename]]\s++[[nspaces:nodename]]\s++[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[nspaces:bytes]?[[all:other]]

1 Solution

somesoni2
SplunkTrust
SplunkTrust

Based on the sample logs you shared in your prev question.

varnishncsa bal-1234 1.48.1.2 - - [22/Aug/2014:15:04:45 +0000] "GET http://www.test.com/error HTTP/1.1" 404 30041 "http://www.test.com/test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2" 0.090000868 miss pass request_id="v-a732c002-2a0d-11e4-88b7-12313d2d8c3b" "-"

try adding following to your sourcetype definition in props.conf

[acquia_varnish_log]
...other entries...
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z 
TIME_PREFIX = \[

Update:

The entries looks good to me. Can try this workaround (setup calculated fields )

[acquia_varnish_log]
    ...other entries...
EVAL-date_hour = strftime(_time,"%H")
EVAL-date_mday = strftime(_time,"%d")
EVAL-date_minute = strftime(_time,"%M")
EVAL-date_month = strftime(_time,"%H")
EVAL-date_second = strftime(_time,"%S")
EVAL-date_wday = strftime(_time,"%A")
EVAL-date_year = strftime(_time,"%Y")
EVAL-date_zone = strftime(_time,"%z")

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Based on the sample logs you shared in your prev question.

varnishncsa bal-1234 1.48.1.2 - - [22/Aug/2014:15:04:45 +0000] "GET http://www.test.com/error HTTP/1.1" 404 30041 "http://www.test.com/test" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.78.2 (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2" 0.090000868 miss pass request_id="v-a732c002-2a0d-11e4-88b7-12313d2d8c3b" "-"

try adding following to your sourcetype definition in props.conf

[acquia_varnish_log]
...other entries...
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z 
TIME_PREFIX = \[

Update:

The entries looks good to me. Can try this workaround (setup calculated fields )

[acquia_varnish_log]
    ...other entries...
EVAL-date_hour = strftime(_time,"%H")
EVAL-date_mday = strftime(_time,"%d")
EVAL-date_minute = strftime(_time,"%M")
EVAL-date_month = strftime(_time,"%H")
EVAL-date_second = strftime(_time,"%S")
EVAL-date_wday = strftime(_time,"%A")
EVAL-date_year = strftime(_time,"%Y")
EVAL-date_zone = strftime(_time,"%z")

somesoni2
SplunkTrust
SplunkTrust

Thanks @kristian for pointing that out. Just updated the same.

0 Karma

kristian_kolb
Ultra Champion

regex is not supported for TIME_FORMAT. Change the '\s' for a ' ', i.e. a blank.

/k

dhavamanis
Builder

Thanks, I have tried adding the TIME_FORMAT, but still its not extracting the required fields. please correct me if anything wrong.

[acquia_varnish_log]
MAX_TIMESTAMP_LOOKAHEAD = 128
REPORT-access = acquia-access-extractions
SHOULD_LINEMERGE = False
TIME_FORMAT = %d/%b/%Y:%H:%M:%S\s%z
TIME_PREFIX = \[
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...