So I have the following data in the log file and would like to be able to search on the specific field name, but unfortunately the data is not in key=value or key:value it's in {"key","value"} format.
Is there any way to do field extraction or a dynamic rex statement to auto-fill the fieldname slot?
Example Data set #1:
{"type","voice"}
{"ani","4561585649"}
{"dnis","956165165"}
{"primary_ani","456159"}
{"primary_dnis","4565657"}
{"loginid","556467db000000000a5c343c45aac2"}
{"agent_key",""}
{"voice_direction","inbound"}
{"calltype","direct"}
{"ctype","direct"}
{"ucid","00002156156564451418"}
{"vdu_id","556467db000000000a5c343c45aac2"}
{"owner","556467db000000000a5c343c45aac2"}
{"createtime","2014-08-12 16:36:58"}
{"createtimet","1407961859"}
{"phone",""}
{"dest","5555555"}
{"orig","91584786594"}
{"ext","55555555"}
{"agent",""}
{"agent.1","556467db000000000a5c343c45aac2"}
Example Data set #2:
{"Comp.adjunct.+",""}
{"Comp.Abandon","N"}
{"Comp.ScreenPopType","None"}
{"Comp.adjunct..leg_id","6"}
{"Comp.adjunct..origvdn","5658978"}
{"Comp.adjunct..destvdn","1589586"}
{"Comp.adjunctcount","6"}
{"Comp.adjunct..report_name","NULL"}
{"Comp.CalltypeCategory","Cust_Service"}
{"Comp.Region","Midwest"}
{"Comp.destvdn","5698578"}
{"Comp.adjunct..dnis","1548659"}
{"Comp.adjunct..menuname"," "}
{"Comp.SurveyCode",""}
{"Comp.adjunct..workflow","dept.calltype_route"}
{"Comp.adjunct..calltype","english_ea_general"}
{"Comp.adjunct..staffinggroup","AAAA"}
{"Comp.adjunct..calltypecategory","Cust_Service"}
{"Comp.CallType","english_ea_general"}
{"Comp.StaffingGroup","AAAA"}
{"Comp.ANI","5847859356"}
{"uui","1589648795,,,5689475,1"}
{"Comp.language","EN"}
{"Comp.todcheck","GENERAL"}
{"Comp.adjunct..timestamp","2014-08-12 11:28:08.681"}
Since I cannot guarantee the order or that all the fields will be present the only way I can think of achieving this is:
index=text
| fields _raw
| rex field=_raw "\{\"type\",\"(?<type>.*?)\"\}"
| rex field=_raw "\{\"primary_ani\",\"(?<primary_ani>.*?)\"\}"
| rex field=_raw "\{\"dest\",\"(?<dest>.*?)\"\}"
| rex field=_raw "\{\"orig\",\"(?<orig>.*?)\"\}"
| where type="voice" AND dest!="" | table *
You have one option which to create a transform for your sourcetype. Below is a props and transfroms created from your sample data. I've tested this against the sample and it works, milage may vary.
#props.conf
[foobarsourcetype]
REPORTS-spam = spam
#transforms.conf
[spam]
SOURCE_KEY = _raw
REGEX = {"([^"]+)","([^"]+)"}
FORMAT = $1::$2
Hopes this helps.
Cheers,
Define a transforms.conf stanza like this:
[some_key_value_name]
REGEX = \{"(?<_KEY_1>[^"]+)","(?<_VAL_1>[^"]+)"\}
Apply that your your sourcetype in props.conf like this:
[your_sourcetype]
...
REPORT-foo = some_key_value_name
Those two magic field names handle extraction of both the field name and field value in one go. See http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/transformsconf for reference.
Note, my regex expects no escaped double quotes in either the field name or value. If that can happen then you need to be a bit more careful with the expression.