Splunk Search

How to autofill field name with field extraction or dynamic rex statement?

steven10172
Explorer

So I have the following data in the log file and would like to be able to search on the specific field name, but unfortunately the data is not in key=value or key:value it's in {"key","value"} format.

Is there any way to do field extraction or a dynamic rex statement to auto-fill the fieldname slot?

Example Data set #1:

{"type","voice"}
{"ani","4561585649"}
{"dnis","956165165"}
{"primary_ani","456159"}
{"primary_dnis","4565657"}
{"loginid","556467db000000000a5c343c45aac2"}
{"agent_key",""}
{"voice_direction","inbound"}
{"calltype","direct"}
{"ctype","direct"}
{"ucid","00002156156564451418"}
{"vdu_id","556467db000000000a5c343c45aac2"}
{"owner","556467db000000000a5c343c45aac2"}
{"createtime","2014-08-12 16:36:58"}
{"createtimet","1407961859"}
{"phone",""}
{"dest","5555555"}
{"orig","91584786594"}
{"ext","55555555"}
{"agent",""}
{"agent.1","556467db000000000a5c343c45aac2"}

Example Data set #2:

{"Comp.adjunct.+",""}
{"Comp.Abandon","N"}
{"Comp.ScreenPopType","None"}
{"Comp.adjunct..leg_id","6"}
{"Comp.adjunct..origvdn","5658978"}
{"Comp.adjunct..destvdn","1589586"}
{"Comp.adjunctcount","6"}
{"Comp.adjunct..report_name","NULL"}
{"Comp.CalltypeCategory","Cust_Service"}
{"Comp.Region","Midwest"}
{"Comp.destvdn","5698578"}
{"Comp.adjunct..dnis","1548659"}
{"Comp.adjunct..menuname"," "}
{"Comp.SurveyCode",""}
{"Comp.adjunct..workflow","dept.calltype_route"}
{"Comp.adjunct..calltype","english_ea_general"}
{"Comp.adjunct..staffinggroup","AAAA"}
{"Comp.adjunct..calltypecategory","Cust_Service"}
{"Comp.CallType","english_ea_general"}
{"Comp.StaffingGroup","AAAA"}
{"Comp.ANI","5847859356"}
{"uui","1589648795,,,5689475,1"}
{"Comp.language","EN"}
{"Comp.todcheck","GENERAL"}
{"Comp.adjunct..timestamp","2014-08-12 11:28:08.681"}

Since I cannot guarantee the order or that all the fields will be present the only way I can think of achieving this is:

index=text
| fields _raw
| rex field=_raw "\{\"type\",\"(?<type>.*?)\"\}"
| rex field=_raw "\{\"primary_ani\",\"(?<primary_ani>.*?)\"\}"
| rex field=_raw "\{\"dest\",\"(?<dest>.*?)\"\}"
| rex field=_raw "\{\"orig\",\"(?<orig>.*?)\"\}"
| where type="voice" AND dest!="" | table *
0 Karma

bmacias84
Champion

You have one option which to create a transform for your sourcetype. Below is a props and transfroms created from your sample data. I've tested this against the sample and it works, milage may vary.


#props.conf
[foobarsourcetype]
REPORTS-spam = spam


#transforms.conf
[spam]
SOURCE_KEY = _raw
REGEX = {"([^"]+)","([^"]+)"}
FORMAT = $1::$2

Hopes this helps.

Cheers,

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Define a transforms.conf stanza like this:

[some_key_value_name]
REGEX = \{"(?<_KEY_1>[^"]+)","(?<_VAL_1>[^"]+)"\}

Apply that your your sourcetype in props.conf like this:

[your_sourcetype]
...
REPORT-foo = some_key_value_name

Those two magic field names handle extraction of both the field name and field value in one go. See http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/transformsconf for reference.

Note, my regex expects no escaped double quotes in either the field name or value. If that can happen then you need to be a bit more careful with the expression.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...