Splunk Search

How to autofill field name with field extraction or dynamic rex statement?

steven10172
Explorer

So I have the following data in the log file and would like to be able to search on the specific field name, but unfortunately the data is not in key=value or key:value it's in {"key","value"} format.

Is there any way to do field extraction or a dynamic rex statement to auto-fill the fieldname slot?

Example Data set #1:

{"type","voice"}
{"ani","4561585649"}
{"dnis","956165165"}
{"primary_ani","456159"}
{"primary_dnis","4565657"}
{"loginid","556467db000000000a5c343c45aac2"}
{"agent_key",""}
{"voice_direction","inbound"}
{"calltype","direct"}
{"ctype","direct"}
{"ucid","00002156156564451418"}
{"vdu_id","556467db000000000a5c343c45aac2"}
{"owner","556467db000000000a5c343c45aac2"}
{"createtime","2014-08-12 16:36:58"}
{"createtimet","1407961859"}
{"phone",""}
{"dest","5555555"}
{"orig","91584786594"}
{"ext","55555555"}
{"agent",""}
{"agent.1","556467db000000000a5c343c45aac2"}

Example Data set #2:

{"Comp.adjunct.+",""}
{"Comp.Abandon","N"}
{"Comp.ScreenPopType","None"}
{"Comp.adjunct..leg_id","6"}
{"Comp.adjunct..origvdn","5658978"}
{"Comp.adjunct..destvdn","1589586"}
{"Comp.adjunctcount","6"}
{"Comp.adjunct..report_name","NULL"}
{"Comp.CalltypeCategory","Cust_Service"}
{"Comp.Region","Midwest"}
{"Comp.destvdn","5698578"}
{"Comp.adjunct..dnis","1548659"}
{"Comp.adjunct..menuname"," "}
{"Comp.SurveyCode",""}
{"Comp.adjunct..workflow","dept.calltype_route"}
{"Comp.adjunct..calltype","english_ea_general"}
{"Comp.adjunct..staffinggroup","AAAA"}
{"Comp.adjunct..calltypecategory","Cust_Service"}
{"Comp.CallType","english_ea_general"}
{"Comp.StaffingGroup","AAAA"}
{"Comp.ANI","5847859356"}
{"uui","1589648795,,,5689475,1"}
{"Comp.language","EN"}
{"Comp.todcheck","GENERAL"}
{"Comp.adjunct..timestamp","2014-08-12 11:28:08.681"}

Since I cannot guarantee the order or that all the fields will be present the only way I can think of achieving this is:

index=text
| fields _raw
| rex field=_raw "\{\"type\",\"(?<type>.*?)\"\}"
| rex field=_raw "\{\"primary_ani\",\"(?<primary_ani>.*?)\"\}"
| rex field=_raw "\{\"dest\",\"(?<dest>.*?)\"\}"
| rex field=_raw "\{\"orig\",\"(?<orig>.*?)\"\}"
| where type="voice" AND dest!="" | table *
0 Karma

bmacias84
Champion

You have one option which to create a transform for your sourcetype. Below is a props and transfroms created from your sample data. I've tested this against the sample and it works, milage may vary.


#props.conf
[foobarsourcetype]
REPORTS-spam = spam


#transforms.conf
[spam]
SOURCE_KEY = _raw
REGEX = {"([^"]+)","([^"]+)"}
FORMAT = $1::$2

Hopes this helps.

Cheers,

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Define a transforms.conf stanza like this:

[some_key_value_name]
REGEX = \{"(?<_KEY_1>[^"]+)","(?<_VAL_1>[^"]+)"\}

Apply that your your sourcetype in props.conf like this:

[your_sourcetype]
...
REPORT-foo = some_key_value_name

Those two magic field names handle extraction of both the field name and field value in one go. See http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/transformsconf for reference.

Note, my regex expects no escaped double quotes in either the field name or value. If that can happen then you need to be a bit more careful with the expression.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...