topic How to autofill field name with field extraction or dynamic rex statement? in Splunk Search

How to autofill field name with field extraction or dynamic rex statement?

steven10172 — Tue, 12 Aug 2014 21:55:40 GMT

So I have the following data in the log file and would like to be able to search on the specific field name, but unfortunately the data is not in key=value or key:value it's in {"key","value"} format.

Is there any way to do field extraction or a dynamic rex statement to auto-fill the fieldname slot?

Example Data set #1:

{"type","voice"}
{"ani","4561585649"}
{"dnis","956165165"}
{"primary_ani","456159"}
{"primary_dnis","4565657"}
{"loginid","556467db000000000a5c343c45aac2"}
{"agent_key",""}
{"voice_direction","inbound"}
{"calltype","direct"}
{"ctype","direct"}
{"ucid","00002156156564451418"}
{"vdu_id","556467db000000000a5c343c45aac2"}
{"owner","556467db000000000a5c343c45aac2"}
{"createtime","2014-08-12 16:36:58"}
{"createtimet","1407961859"}
{"phone",""}
{"dest","5555555"}
{"orig","91584786594"}
{"ext","55555555"}
{"agent",""}
{"agent.1","556467db000000000a5c343c45aac2"}

Example Data set #2:

{"Comp.adjunct.+",""}
{"Comp.Abandon","N"}
{"Comp.ScreenPopType","None"}
{"Comp.adjunct..leg_id","6"}
{"Comp.adjunct..origvdn","5658978"}
{"Comp.adjunct..destvdn","1589586"}
{"Comp.adjunctcount","6"}
{"Comp.adjunct..report_name","NULL"}
{"Comp.CalltypeCategory","Cust_Service"}
{"Comp.Region","Midwest"}
{"Comp.destvdn","5698578"}
{"Comp.adjunct..dnis","1548659"}
{"Comp.adjunct..menuname"," "}
{"Comp.SurveyCode",""}
{"Comp.adjunct..workflow","dept.calltype_route"}
{"Comp.adjunct..calltype","english_ea_general"}
{"Comp.adjunct..staffinggroup","AAAA"}
{"Comp.adjunct..calltypecategory","Cust_Service"}
{"Comp.CallType","english_ea_general"}
{"Comp.StaffingGroup","AAAA"}
{"Comp.ANI","5847859356"}
{"uui","1589648795,,,5689475,1"}
{"Comp.language","EN"}
{"Comp.todcheck","GENERAL"}
{"Comp.adjunct..timestamp","2014-08-12 11:28:08.681"}

Since I cannot guarantee the order or that all the fields will be present the only way I can think of achieving this is:

index=text
| fields _raw
| rex field=_raw "\{\"type\",\"(?<type>.*?)\"\}"
| rex field=_raw "\{\"primary_ani\",\"(?<primary_ani>.*?)\"\}"
| rex field=_raw "\{\"dest\",\"(?<dest>.*?)\"\}"
| rex field=_raw "\{\"orig\",\"(?<orig>.*?)\"\}"
| where type="voice" AND dest!="" | table *

Re: How to autofill field name with field extraction or dynamic rex statement?

martin_mueller — Tue, 12 Aug 2014 23:00:00 GMT

Define a transforms.conf stanza like this:

[some_key_value_name]
REGEX = \{"(?<_KEY_1>[^"]+)","(?<_VAL_1>[^"]+)"\}

Apply that your your sourcetype in props.conf like this:

[your_sourcetype]
...
REPORT-foo = some_key_value_name

Those two magic field names handle extraction of both the field name and field value in one go. See http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/transformsconf for reference.

Note, my regex expects no escaped double quotes in either the field name or value. If that can happen then you need to be a bit more careful with the expression.

Re: How to autofill field name with field extraction or dynamic rex statement?

bmacias84 — Tue, 12 Aug 2014 23:11:50 GMT

You have one option which to create a transform for your sourcetype. Below is a props and transfroms created from your sample data. I've tested this against the sample and it works, milage may vary.



#props.conf

[foobarsourcetype]

REPORTS-spam = spam



#transforms.conf

[spam]

SOURCE_KEY = _raw

REGEX  = {"([^"]+)","([^"]+)"}

FORMAT = $1::$2

Hopes this helps.

Cheers,