All Apps and Add-ons

Regex not working in props.conf as per when searched with Rex command

Builder

I have created a regex that works fine during search time, but when added to props.conf and/or transforms.conf to extract the field during index time, the field doesnt get extracted?

I dont understand how this could work during search time in the Splunk Search bar search page, but not when added to props.conf?

Here it is:

rex field=_raw "set=(?<phoneid>.+)\snotTime"
0 Karma

Splunk Employee
Splunk Employee

^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),

Motivator

please stop posting comments as new answers. thanks.

Explorer

i tried with the props only and i still cannot see the fields. has this anything to do with splunk 6.0.3. the other colleague of mine created field extraction and does not see them as well. it was ok two weeks ago before upgrade to splunk 6.0.3

0 Karma

Splunk Employee
Splunk Employee

props.conf
[sdf_bpel_metric]
REPORT-sdf_policy_metric = SDFCorepolicymetrics

transforms.conf
[SDFCorepolicymetrics]
REGEX = ^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),

0 Karma

Splunk Employee
Splunk Employee

props.conf only
EXTRACT-SDFCorepolicymetrics = ^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),

0 Karma

Splunk Employee
Splunk Employee
  • | rex field=_raw "^(?:.+?,){4}(?.+?),(?.+?),.+?,(?.+?),(?.*?),(?.+?),"
0 Karma

Splunk Employee
Splunk Employee
0 Karma

Explorer

requestApplicationLabel [40-52] MetricLogger

requestTransactionID [53-67] TDI_CLOUDCSX_1

callingApplication [82-115] hymlxsdfbpe11_1401889362113_11537

callType [116-116] ``

function [117-140] RetrieveIdentityDetails

second log

requestApplicationLabel [63-75] MetricLogger

requestTransactionID [76-111] TELSTRA_PREPAIDACTIVATION_STRATEGIC

callingApplication [148-180] chslxsdfbpe05_1401889356427_2871

callType [181-181] ``

function [182-212] CCandB.CreateNewBillingAccount

i got this from regex101.com. and tested in search field in splunk. it was ok. unless it works differently?

0 Karma

Explorer

yep. here are two sample logs

2014-06-04 23:42:42,115,,,1401889361349,MetricLogger,TDI_CLOUDCSX_1,1401889361349,hymlxsdfbpe11_1401889362113_11537,,RetrieveIdentityDetails,148

2014-06-04
23:42:36,427,,,0dedf85a-fbdb-43cb-b9f1-d4a0f636ab97,MetricLogger,TELSTRA_PREPAIDACTIVATION_STRATEGIC,0dedf85a-fbdb-43cb-b9f1-d4a0f636ab97,chslxsdfbpe05_1401889356427_2871,,CCandB.CreateNewBillingAccount,2983

i tried two methods,
FIRST method, just in props as below. does not quite work. worked when i use rex field=_raw "regex" though in search field though. tested in on one of those regex online as well

[sdf_bpel_metric]
EXTRACT-SDFCorepolicymetrics = (?:[^,\n],){5}(?P[a-zA-Z]+),(?P[^,]),(?:[^,\n],)(?P[^,]),(?P[^,]),(?P[^,])

SECOND method

in props

[sdf_bpel_metric]
REPORT-sdf_policy_metric = SDFCorepolicymetrics

in transforms

[SDFCorepolicymetrics]
FORMAT = requestApplicationLabel::$1 requestTransactionID::$2 callingApplication::$4 callType::$5 function::$6
REGEX = ([a-zA-Z]+),([^,]),([^,]),([^,]),([^,]),([^,]*),

0 Karma

Ultra Champion

And perhaps of the (relevant portions of) props.conf, and perhaps inputs.conf as well (only the portion where you configure the input of this file).

0 Karma

Splunk Employee
Splunk Employee

Can you provide a sample of the raw log?

0 Karma

Ultra Champion

This may indicate that the EXTRACT is not applied at all. Under what stanza header have you put the EXTRACT? Does this match the sourcetype/source/host?

Legend

Just want to point out that you don't need to reingest the log, or restart Splunk. Field extractions happen (mostly) at search-time, regardless of if they happen in props.conf/transforms.conf or inline in your search.

0 Karma

Builder

Yes, thats exactly what I have set and it doesnt work, no matter how much I perform a restart or Log Re-ingestion.

0 Karma

Champion

what's your props.conf settings? Is it the below or not?

EXTRACT-PHID= set=(?<phoneid>.+)\snotTime

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!